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ABSTRACT 

This casebook provides a reference and technical 
assistance tool for citizens and consumers with varying levels of 
knowledge or experience in the field of evaluation and monitoring, 
focusing on mental health centers. The purposes of this casebook and 
current knowledge about citizen evaluation are discussed. The 51 case 
summaries presented are grouped into these categories: (1) the 
evaluator*s place in the mental health system; (2) extent of 
involvement of evaluators; (3) scope of coverage of evaluation; (4) 
focus of evaluation; and (5) procedures for evaluation. The format 
for case studies consists of a summary, type of organization 
evaluated, description of evaluators, reasons for evaluation, level 
of participation, target of evaluation, problems or issues evaluated, 
techniques used, findings of evaluation activities, recommendations, 
steps to ensure implementation, extent of implementation, special 
barriers to or support for evaluation, resources and costs, 
additional comments, and contact person. References and a glossary of 
terms are included. (ABL) 
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Foreword 



Citizen evaluation represents a confluence of two movements: citi- 
zen participation and program evaluation. Citizen participation is a move 
to empower the public, and particularly the intended targets of programs, 
by providing opportunities for citizens and clients to guide programs to- 
ward their needs. Program evaluation is an effort to develop methods to 
study how well a program accomplishes what it is supposed to do. One 
goal of both these movements is to increase accountability of programs to 
the public and program funders. In citizen evaluation, lay citizens and 
consumers evaluate programs as one way to achieve the goal of program 
accountability. 

The National Institute of Mental Health (NIMH) has supported both 
the citizen participation and program evaluation movements. In the Com- 
munity Mental Health Centers (CMHC) program specific requirements in 
the Amendments of 197! (Public Law 94-63) advanced the cause of 
citizen participation for r.^sidents of each CMHC catchment area in the 
governance of the local mental health center. CMHCs were also required 
to report their service statistics and to conduct program evaluations on 
designated topics. ITus law, furthermore, required each center to engage 
in citizen evaluation of a program review type, namely, each center 

will, in consultation with residents of its catchment area, 
review its program of services and the statistics and other 
[evaluative] information ... to assure that its services are 
responsive to the needs of the residents of the catchment area. 

NIMH has attempted to assist CMHCs and catchment area residents 
in carrying out these requirements in a variety of ways. Initial efforts 
focused on manuals (Hagedom et al. 1976), resource materials (e.g., Har- 
greaves et al. 1977), and workshops for program evaluation specialists and 
administrators. Pertinent materials were produced for the varioiis aspects 
of evaluation and quality assurance carried out by clinical peers (Hage- 
dom et al. 1976) and site visit monitoring by Government funding agen- 
cies (NIMH 1979). As CMHCs developed experience In program evaluation 
it became possible to produce casebooks (Landsberg et al. 1979; Gabbay 
and Windle 1975) that illustrate what some CMHCs have been able to do 
in practice. While these materials are of some help to lay citizen groups, 
material more specifically tailored to their backgrounds, perspectives, 
and purposes seems more useful. Several manuals have been developed to 
orient and guide citizen groups in program evaluation, some developed by 
NIMH (Peters et al. 1980), some by research groups with NIMH grants to 
work on citizen evaluation (Zinober ;\nd Dinkel 1981; MacMurray et al. 
1976), and some by citizen groups themselves (Green and Matthews 1979; 
Mental Health Association 1976). 

This Casebook builds on this sequence of technical assistance, moving 
from general manuals to specific cases by providing examples of how 
citizen groups have been able to put program evaluation into practice. 
The project to develop this Casebook was proposed by the NIMH Regional 
Office Task Force on Program Evaluation, endorsed by the National 
Council of Community Mental Health Centers and the Council on Re- 
search and Evaluation, and financed as part of NIMH's technical assis- 
tance to CMHCs. This Casebook was designed initially to parallel an ear- 
lier casebook, Evaluation in Practice: A Sourcebook of Program Evalua- 
tion Studies from Mental Health Care Systems in the United States 

iii 
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(Landsberg et al. 1979), but differences in the scope and approaches of 
citizen evaluations prompted considerable variation in the report outline. 
The present Casebook shows diversity in the forms of evaluations that 
citizen groups have attempted, variety in the methods used, and different 
results from these various experiences and approaches. What the collec- 
tive and cumulative results of these experiences will be are in the process 
of being determined as citizen gioups around the country reflect on their 
own experiences and on the experiences reported to them by others. The 
present Casebook, while not designed specifically to document represen- 
tative evaluation experiences, can help citizen groups become aware of 
the breadth of evaluative activities in the late 1970s and early 1980s. The 
organization of the Casebook, which draws attention to the importance of 
using the evaluation results and clarifies other aspects of the evaluation 
process, will be especially helpful to citizen evaluators. 

J Human Services Research Institute (HSRI) has done an excellent 
job in drawing on past NIMH projects and publications and HSRI's own 
past work with citizen and advocacy org;»nizations and has been able to 
locate new cases veuying widely in settings and approaches and to col- 
laborate with these citizen evaluators in developing standardized but in- 
teresting descriptions of these experiences. The result is that this Case- 
book blends technical mformation and the sense of actually being in a 
program evaluation as citizen groups attempt to make improvements 
through this emergir.g mechanism. NIMH is pleased to have supported the 
development of this important work. 



I.emuel Clark, M.D. 
James Stockdill 

Office of State and Community Liaison, NIMH 
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Purpose of the Casebook 



Objectives and Goals 

The major objective of this Casebook is to 
provide a reference and technical assistance tool 
for citizens and consumers with varying levels of 
knowledge or experience in the field of evaluation 
and monitoring. For example, the Casebook should 
be as useful to the novice citizen evaluator who 
needs to understand the variety of approaches 
that are available in evaluation or monitoring as 
to the advanced citizen evaluator who may simply 
need additional techniques or implementation 
strategies. In order to meet this primsuy objec- 
tive, the Casebook seeks to highlight the breadth 
and diversity of citizen and consumer evaluation 
ana monitoring activities in mental health and 
other human services across the country. 

More specific objectives include the following: 

c To provide concrete examples of citizen 
participation in independent evaluations and 
in the review of agency mental health eval- 
uation plans and results 

• To differentiate evaluation activities ac- 
cording to the nature of citizen and consiimer 
organizations (e.g., boards, free-standing en- 
tities, etc.), and their regulatory, statutory, 
and/or organizational purposes 

• To display the multiple means available to 
citizens and consumers in program evaluation 

• To organize citizen and consumer evaluation 
experience in a coherent fashion 

• To prepare a document that challenges cit- 
izens and consumers to stretch ttieir capa- 
bilities and "demystifies" the area of program 
evaluation 

• To point out the ways in which program 
evaluation can be used to bring about service 
improvement and reform 

• To illustrate certain pitfalls that may occur 
during the evaluation process and that should 
be avoided 

• To show mental health administrators, as 
well as citizens and consumers, the positive 
benefits of citizen participation in program 
evaluation 



• To encourage citizens and consumers to un- 
dertake the task of evaluation with a real- 
istic appreciation of its feasibility, benefits, 
and difficulties 

Finally, and most importantly, the Casebook 
will provide citizens and consumers with in- 
creased confidence in their capabilities and power 
to influence the shape and content of mental 
health services. 



Preparing the Casebook 
Project Conception 

Tne community mental health movement con- 
tained concerns for program evaluation, citizen 
participation, and the combination of these pro- 
cedures for program guidance and citizen eval- 
uation. These concerns took the form of specific 
requirements in the Community Mental Health 
Center (CMHC) program (Public Law 94-63). 
During the implementation of the CMHC pro- 
gram, the National Institute of Mental Health 
(NIMH) followed a strategy of technical assist- 
ance in program evaluation that began by focusing 
on the development of manuals, orientation con- 
ferences, and training for CMHC professional 
staff. The second step was manuals for citizens 
and casebooks for professionals. The third step 
was casebooks and audiovisual orientation mater- 
ial for lay or citizen evaluators (several of these 
manuals are described in the introduction to the 
section, Wkat Do We Know About Citizen Evalu- 
ation and Monitoring). 

In order to address the specific needs and in- 
terests of citzerts and consumers, NIMH staff 
commissioned a publication to illustrate the range 
of evaluation and monitoring activities that cit- 
izen groups can perform. The result is this Case- 
book, which can be seen as (1) a companion vol- 
ume to NIMH's Sourcebook of Program Evaluation 
Studies from Mental Health Care Systems in the 
United States (Landsberg et al. 1979), hereafter 
called Sourcebook; and (2) a component of a 
larger body of information available to citizens 
and consumers. For instance, instead of repeating 
the various approaches to evaluation developed in 
another NIMH-sponsored publication Citizen 
Roles in Community Mental Health Center Eval- 
uation, this Casebook shows citizens and 
consumers how these various approaches have 
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acUtally been applied in concrete situations. In 
addition, those readers who want more informa- 
tion on the numerous evaluation methods and in- 
struments available in the field can refer to the 
Soitrcebook. In other words, this Casebook is ?art 
of a sequence of technical assistance mate- 
rials—the components of which are internally 
consistent in direction and useful in specific and 
unique ways. 

Project Initiation 

The development of the Casebook was aided 
greatly by NIMH project staff and members of the 
Project Advisory Committee composed of citi- 
zens, consumers, and professional evaluators. 
These two groxxps assisted HSRl staff in making 
key decisions regarding the types of citizen in- 
volvement to include in the Casebook, the def- 
initions of evaluation and monitoring activities, 
the criteria to be used in selecting examples of 
citizen evaluation, and the methods by which such 
examples would be obtained. 

In order to make the Casebook relevant to the 
widest possible audience and to encourage in- 
creased participation in evaluation by a broad 
range of individuals, the term "citizen involve- 
ment" is defined quite broadly. '^Citizens" in this 
context are individuals who are external to the 
day-to-day operation of the system, agency, or 
program being evaluated, who may or may not 
have special expertise in evaluation research or 
mental health systems administration, and who 
are usually acting in a volunteer, unpaid capacity. 
"Involvement" in evaluation or monitoring covers 
every step in the process, from initiating an 
evaluation or monitoring program througji devel- 
oping recommendations based on in-house or 
professional evaluations and devising implemen- 
tation strategies based on evaluation results. 

Next, an overall definition of evaluation and 
monitoring had to be developed that encompassed 
pure evaluation activities (e.g., those with proven 
reliability and validity), as well as interesting and 
important examples that did not necessarily meet 
strict methodological tests. Hagedom et al.'s 
(1976) definition captured this spirit: 

In its most basic form, evaluation is a com- 
parison of actual program operations and 
results against a standard. The standard of 
comparison is usually data from similar pro- 
grams or a judgment by an accepted author- 
ity or group on what should be expected. 

With tins as the core, the definition was expanded 
to include judgments by clients regarding agency 
performance and judgments based on norms- 
either ideological or professional. 

Given this basic formula, certain activities 
were eliminated from consideration. For example, 
certain activities such as dissemination of in- 
formation regarding center operations fell outside 



the definition of evaluation. Other efforts, such 
as the conduct of public forums to gain com- 
munity input regarding service acceptability and 
accessibility, were incorporated within the 
definition. 

Project staff and others also determined that 
the scope of tl*e Casebook should be expanded to 
include monitoring activities. Monitoring is dif- 
ferent from evaluation in that monitoring tends to 
be ongoing and involves observations and review 
of secondary data. Evaluation is more Ukely to be 
limited in duration and ro involve the generation 
of primary data. These distinctions, however, are 
not hard and fast. Based on feedback from citi- 
zens and professionals involved in these activities, 
it became evident to project staff that monitoring 
bodies, such as judicially appointed compliance 
mechanisms and m^rsing home review groups, 
were performing activites that reqidre techniques 
similar to those employed in evaluation. Likevdse, 
monitoring has a related and usually positive in- 
fluence on service quality. 

In additic i, specific criteria were developed to 
guide the selection of cases. The following list of 
criteria were applied to each case: 

• Transferability- Does the example reflect 
efforts that can, with relative ease, be 
transferred to similar contexts? 

• Use of results- -Does the case represent ac- 
tivities of direct use in either impro\rtng an 
aspect of a mental health service facility or 
achieving general system improvement? 

• Practicality- -Does the case contain methods 
^hat are relatively easy to apply and that do 
not require substantial training, funding, or 
personnel resources? 

• Impan- Does that case reflect some impact 
or outcome such as agency improvement, 
increased citizen interest, or reorientation of 
agency priorities? 

• Diversity— Doc^ the case contribute to a 
range of evaluation methods, settings, and 
auspices? 

• Validity- - Does the measurement process ad- 
equately represent the phenomenon being 
assessed? 

• Acceptability- Does the case reflect tech- 
niques and goals that are generally accept- 
able and likely to be adopted by consumers, 
citizens, and ultimately policymakers and 
administrators? 

In order to ensure that the citizen perspective 
was represented in the case descriptions, inter- 
ested citizens or consumers were asked to write 
up cases where at all possible, in those instances 



where this was not feasible, every effort was 
made to include at least some consumer or citizen 
involvement in the preparation of the case 
sununary. 

Finally, project staff of the Human Services 
Research Institute (HSRI) employed ntimerou? 
techniques to solicit the case examples: 

• A brief description of the project was pre- 
pared and sent to more than 35 national or- 
ganizations and professional associations 
concerned with mental health or mental 
disabilities, incltading the National Council of 
Community Mental Health Centers, the Men- 
tal Health Association, and generic citizen 
groups such as the Center for Responsive 
Governance. At least 16 of these organiza- 
tloD% included the project deccription in their 
association newsletters or other periodicals. 
The National Association of State Mental 
Health Program Directors sent the project 
description to each of the 50 State depart- 
ments of mental health. This action alone 
yielded 31 responses. 

• A direct mailing was sent to the respondents 
of the survey that was part of the citizen 
evaluation project that produced A Trust of 
Evaluation (described on page 10), These re- 
spondents were members of either mental 
health associations or community mental 
heaith center boards. Approximately 21 re- 
sponses were obtained from this strategy. 
This mailing was conducted by Joan Zinober 
who also conducted the initial survey. In this 
way, confidentiality was not violated. 

• A number of knowledgeable persons at the 
Federal, State, and local levels familiar with 
e;:emplary cases involvifig citizen evaluation 
were contacted. For example, a member of 
the Project Advisory Committee distributed 
the project description to the chairpersons of 
the 59 California County Mental Health Ad- 
visory Boards. 

• Several reference materials secured from 
association newsletters also provided a cer- 
tain number of potential cases for the 
Casebook. 

HSR! staff received a total of 126 responses to 
the case solicitation and selected 93 as potential 
cases. They telephoned each potential respondent 
and followed up with a letter confirming lus oi 
her l*"terest in participating in the Casebook. 
Specific materials explaining the format for the 
cases were also provided to authors. 

The final Casebook contains 51 cases repre- 
senting every region of the country, a variety of 
auspices and target groups, a multiplicity of 
methods, and a range of mental health-related 
issues. In order to capture the richness of each 



evaluation or monitoring experience, the case 
descriptions follow the same format and are or- 
ganized to cover 16 categories of information. 
These categories are described in the format for 
case description sent to all case authors (see box). 



How to Use the Casebook 

The Casebook has been designed to present 
information in a manner that is easy to under- 
stand and may stimulate further interest in 
citizen evaluation and monitoring. 

After considering several different ways of 
organizing the material, HSRI staff— in consul- 
tation with NIMH and the Projoct Advisory Com- 
mittee — decided to present the cases in six major 
chapters that focus on organizational context, 
role, scope, topic, method, and techniques for 
implementing findings. The chapters follow the 
logical sequence of steps that a citizen or con- 
sumer should take to initiate and carry out an 
evaluation or monitoring task. The specific con- 
tent of the chapters is outlined below. 

• Organizational Context — Where Are You in 
the System"^ 

In this section, citizens will learn how to 
examine their position in the larger scheme 
of things. The need to understand where a 
citizen group fits in the system as well as 
what rights aiji responsibilities it has is an 
important first step. The cases in this chap- 
ter show citizens functioning in a variety of 
settings ranging from State mental health 
advisory boards to boards of local agencies. 
These cases should help the reader to deter- 
mine his or her own position In the system 
and the implications that this position has for 
the cesign of an evaluation or monitoring 
activity. 

• Role-'How Much Will You Be Involved'! 

This chapter addresses the importance of 
making a realistic appraisal of the level of 
commitment that can be expected from each 
member and from the group as a whole. This 
appraisal, coupled with the organizational 
concerns in the chapter above, vdll determine 
how active a role tie individual or group will 
play. Cases in this chapter show citizens and 
consumers taking on a range of roles in- 
cluding simply reacting to the resxdts of 
evaluations conducted by others to conduct- 
ing full-scale independent evaluations. 

• Scope: Ho(w Far Will You Cast the Nett 

Evaluation and monitoring activities can fo- 
ciis on a range of targets from very broad to 
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Format for Case Descriptions 



Sunmury 

Type of Organization 
Evaluators or Monitors 



P • * ?or Evah)£ticn 
Level of Participation 



Target of Evaluation 
or Monitoring 



Problems or Issues 
Evaluated or Monitored 



Techniques Used 



Findings of Evaluation 
or Monitoring Activities 



Recommendations 



Steps to Ensure 
Implementation 



Extent of 
Implementation 

Special Barriers to or 
Supports for 
the Evaluation 



Resources and Costs 



Additional Conunenu 



Contact Peraon 



Please provide a brief overview of the evaluation or monitoring activity in order to introdiice 
the reader to your experiences. 

Please tell us what type of group or organization conducted the evaluation or monitoi ing 
activity. Was it a community mental h^^lth .enter board? A State mental health advisory 
board? An advocacy organization? Also, please include any other relevant information on the 
organization including such things as how it is constituted, how many members it has, who 
appoints the members, and what the formal responsibilities of the group are. 

Please tell us something about the Individuals who carried out the project. Were they pro- 
fessionals, lay persons, consumers, civic leader?, or a combination of individuals? 

Please tell us what prompted the evaluation or monitoring activity? Is it mandated by S^-ate 
law? Was a particular problem brought to the attention of ycur organization? Are you part of 
an ongoing monitoring activity? 

Please tell us how involved you were in the evaluation or monitoring activity. There are 
several forms that your participation may have taken. You may have been asked by an agency 
to provide information. You may have reviewed informaticn. prepared by a service agency or 
service system. You may have joined together with an agency to conduct an evaluation or you 
may have conducted an evaluation that was totally independent of the service system. 

Please tell us about the agency or program which you assessed. We need to tao'v two things. 
First, how broad was the suidy— did it encompass a whole State, a count>- or regr^-Al system, 
a comprehensive agency, one aspect of a program, or a cluster of programs? Second, what 
type of agency or program is being evaluated? In responding to the second point, please let us 
know the name of the agency or agencies, the type of community in which it is located, the 
nature of its administrative structure and any other pertinent infomution about the witity 
that you feel is relevant. 

Here, we are interested in learning about the nature of the Issues or problem which you or 
your group addressed during the evaluation or monitoring activity. For instance, were ycu 
concerned about the needs of special groups such as the elderly or children? Were you pri~ 
marily interested in the way the program carried out its administrative resiponsibilities, or 
were you more interested in the physical plant? Were individual rights of clients a prime 
focus or did you focus more generally on service quality.-' 

Please discuss the types of techniques or methods you used to gather information for your 
evaluation or monitoring activity. Techniques might include such things as site visits, public 
forums, community surveys, consumer satisfaction questionnaires, or facility check-lists. 
Please let us know how the techniques were selected, who participated in the design of the 
study, and whether the techniques proved useful. 

Please teU us what you found out as a result of the evaluation or monitoring project. For in- 
stance, did you find that services were adequate or inadequate? Is more funding required? 
Are there certain administrative practices that should be changed or improved? Is there a 
particular segment of the pi*ogram's potential client group that is not being served? 

Based on the findings which you have described above, what did you or your groi^ recommend 
as ways of improving the program or programs you assessed? If the program or agency was 
performing adequately, then let us know what kinds of comments you included in your final 
report. Please also let us know to whom the reconunendations were directed (e.g., the agency 
director, the State legislature, the State department of mental health, etc.). 

Where your recommendations called for some specific change or activity, we are Interested 
in how you went about making sure that these recommendations were a'^nially implemented. 
Did you publish your recommendations in the paper? Did you makt a i -ese* itation to your 
legislature? Did you use the leverage or legal authority of your organize Jon to enforce 
change? 

We are interested in the results of your efforts to secure change or improvement in the 
programs or agency you addressed. Were any or all of your recommendations implemented? If 
not, why not? 

Looking back on your experiences, were there any particular constraints that limited your 
ability to carry out the evaluation or mOiJtoring activity? For instance, did you have problems 
with the technique you used? Were theri political problems that hampered your efforts? Was 
it difficult to secure sufficient time from volunteers to complete the effort? We are also 
interested in a description of those things that assisted you m your assessment s\ich as a 
supportive board of supervisors, a dedicated group of citizen volunteers, or a particularly- 
effective method. 

Please tell us the amount of persoraiel (staff and volunteer) time needed to complete the 
evaluation or monitoring activity. If feasible, please provide costs for any paid staff. It would 
also be interesting to know how you or your group obtained the resources necessary to carry 
out the project. 

Please tell us any further thoughts you may have on your experience. For instance, woxild you 
do it the same way neJCt time? How would you chwige the process? What lessons jld you 
lilcft to share with otheri* in the same circumstances? 

Please list the name, address and phone number of a person familia. with the case Wtio vrill be 
available in the futare to answer any questions. This person may be vf>u or anyone you feel 
would be appropriate as a continuing reference. 
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extreniely narrow. One of the earliest deci- 
sions that groups have to make is the scope 
of their inquiries. This chapter includes cases 
in which citizens evaluated everything from 
an entire system to one component of one 
agency's program. The cases have been se- 
lected to help the reader understand the im- 
portance of keeping the scope of the evalu- 
ation consistent with the group's authority 
and resources. 

• Topic—What WUl Be Your Focus? 

This chapter will help citizens and consumers 
select the issues on which to focus. The cases 
reflect the multiplicity of topics that citizen 
gioups aroimd ttie country have assessed, 
including client rights, administrative prac- 
tices, facilities, and institutional closure. 
These cases should help the reader to select 
topics that are consistent with the mission, 
resources, arid authority of the citizen or 
coiisumer group. 

• Method—Hofw Will You Go About Itl 

This chapter reviews the various evaluation 
and monitoring methods that can be used by 
citizen and consumer organizations. Cases in 
this chapter show citizens applying tech- 
niques as varied as site visits, client inter- 
views, key informant interviews surveys, and 
observation. The examples should help the 
reader to match the area to be explored with 
the most useful means of collecting infor- 
mation about that topic. 



• Implementation — How Will the Results Be 
Used? 

Evaluation or monitoring alone cannot bring 
about change. The results have to reach the 
right hands and must be presented in a form 
that is readily understandable. The impor- 
tance of formulating a plan for circulating 
the results, bringing the Issues to the public's 
attention, and following up on reconutien- 
dations is highlighted by a review of selected 
cases from previous chapters. 

• Summary 

This ijial chapter provides a brief overview 
of the lessons learned from the range of ex- 
periences presented in the cases. 

In order to clarify these steps and to provide 
contimaity for the restder from chapter to chapter, 
a hypothetical or idealized case involving a cit- 
izen evaluator, Doug Brown, has been designed. 
Each chapter begins with the next phase of Doug's 
deliberations as he and his planning and evaluation 
committee grapple vdth key evaluation options 
and considerations. 

The 51 cases in the Casebook have been 
organized under six chapter headings. It should be 
noted, however, that the cases reflect concerns 
beyond the issues raised in any single chapter. 
This cross-reference index has been compiled so 
that the reader can focus not only on one 
particular aspect of the case but can also, through 
the cross-referencing system, determine how it 
fits into the other Casebook categories. 



Cross-Reference Index to Case s^tudies 



Organizaticmal context 

State Council 

Coimty Regional/Board 

Agency(s) Board 
External Group 
Consumer Group 

CMHC Board or 

Subcommittee 
Quality Assurance Board/ 

Board of Visitors 
Advocacy Group 
Patient or Former 

Patient Group 

Role 

Subjects of Evaluation 
Solicited Reactors 



Case number 



1, 22 

2, 3, 6, 16, 21, 24, 25, 29, 
31, 32, 33, 37, 39, 44, 49 

4, 49, 50 

5, 18, 19, 20, 51 

13, 14, 15, 17, 24, 27, 30, 
36, 38, 43, 46, 48 

7, 11, 26, 28, 40, 45, 47 



8, 34, 42 

9, 23, 36 

10, 12, 35, 36 



11, 41 

12. 13, 47, 48 



Partnert Evaluation 



Independent Evaluators 



In-House Evaluators 
External Consultants 

Scope 

State System 
County/Regional System 

Cluster 
Agencywide 



Program Component 



Case nxmber 

1, 2, 3, 6, 7, 10, 14, 15, 16, 
21, 26, 28, 31, 35, 40, 45, 
49 

5, 8, 9, 17, 18, 19, 22, 24, 
25, 27, 29, 30, 32, 33, 34, 
36, 37, 38, 39, 42, 43, 46, 
49 

4, 50 

20 



1, 14, 15, 21, 22 

2, 7, 16, 17, 23, 29, 31, 32. 
33, 39, 44, 49 

5, 9, 19, 30, 34, 36, 38 

3, 4, 6, 8, 10, 11, 12, 18, 20, 
24, 25, 28, 35, 40, 42, 47, 
49, 50 

13, 26. 27. 37. 43. 45. 46. 48 
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Case number 



Case number 



Topic 
Needs 

Administrative Practices 
Facility Concerns 



Funding Resource 

Allocation 
Service Delivery 



Individual Rights 
Client Satisficatlon 

Special Populations 

Service Quality 



Planning 

Acceptability/ Awareness 
Institutional Closure 
Accessibility 

Client Outcomes 



1. 7. 11. 12. 17. 22. 28. 31. 

32. 35. 39. 40. 41. 42. 43. 
47 

3. 5. 6. 15. 16. 18. 20. 21. 
22. 23. 24. 25. 26. 27. 29. 

33. 34. 36. 44. 45, 49 

6. 8. 9. 15. 21. ^13, 25. 27. 

30. 32. 34. 36. 38. 40. 42. 
49 

25. 29. 31. 32. 33. 44 

2. 13. 14. 15. 16. 17. 18. 19. 
20. 21. 23, 24. 25. 26. 28. 

31. 32. 33. 36. 37. 39. 40. 
43, 44. 45. 46. 48. 49. 51 

5. 6. 8. 10. ivs 21. 30. 34 

6. 9. 10. 12. 18. 19. 21. 23. 
26. 35. 37. 48 

1. 25. 27. 28. 30. 36. 37. 45. 
49 

4. 5. 6. 8. 9. 12. 13. 14. 15. 
18. 19. 21. 22. 23. 25. 29. 
35. 36. 37. 38. 39. 42. 45. 
47. 49. 51 

2. 7. 11. 18. 31. 39. 44 

6. 13. 17. 24. 40, 41. 47. 48 
42 

14. 15. 21. 23. 24. 33. 36. 

38. 41. 42, 43. 47. 49 
4. 20. 35, 50 



Method 



Site Visit 



Key Informant 
Interview/S\nvey 



Secondary Data Anal3rsis 



Case Simulation 
Clinician Interview 

Survey 
Stn%;tured Grot^ 

Approaches 
Use of Standardized 

Instruments 
Evaliiability Assessment 
Freedom of 

Information Act 
Client Surve/ 
Client Record Review 
Checklist 
Investigation of 

Grievances 
Computer Modeling 
Public Forum 



2. 6. 7. 8. 14. 15. 19. 20. 
21. 23. 24. 25, 27, 29 30. 
33, 34. 36. 37. 38. 43. 45. 
49. 51 

1. 6. 7. 11. 14. 16. 17. 22. 
24. 28. 29. 30. 39. 4C. 41. 
42. 43. 44. 47 

2. 3. 4. 6. 7. 8. 13. 14. 16. 
18. 20. 22. 25. 26. 27. 28. 
32. 33. 39. 40, 42. 44. 48. 
49. 50, 51 

45 

4, 40, 46. 49 

13. 17. 22. 47. 48 

35. 49 
49, 50. 51 

51 

4. 8. 9. 12. 20. 26. 35 
20. 34. 49 

16. 17. 21. 23. 27. 30. 34. 38 

5. 8. 10. 18. 19. 34 

31 

7. 18. 22 
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What Do We Know About Citizen Evaluation 

and Monitoring? 



Introduction 

Citizen evaluation and monitoring in the field 
of human services generally and mental health 
specifically are relatively new phenomena. Citi- 
zen interest in program oversight has evolved in 
response to several pressures and issues: (1) a 
growing skepticism of Government's and others' 
abilities to provide responsive and efficient 
services; (2) a basic recognition in the United 
States of the value of citizens' ideas; (3) an im- 
petus to eliminate waste, increase efficiency, and 
ensure accountability; (4) a plethora of court 
cases addressing both human rights and client 
rights that cfirectly involve consumers in evalua- 
tion of service effectiveness; (5) use of citizens 
by public agencies to monitor publicly funded 
programs that resist government oversight; and 
(6) involvement of citizens and consumers in order 
to develop support fcr difficult and controversial 
decisions (Dinkel et al. 1982). 

The inclusion of citizens in mental health 
evaluation can be attributed more specifically to 
Federal mandates requiring such participation in 
the CMHC program. The recent enactment of 
block grant legislation (Public Law 97-35), how- 
ever, has removed the Federal statutory re- 
qtdrement for citizen evaluation of CMHCs. De- 
spite this change, citizen evaluation appears to be 
well established as evidenced by the variety of 
examples that can be fomd in the literature, and 
that were solicited for the Casebook. 

For the most part, what we know about citizen 
evaluation and monitoring can be fomd in a small 
but developing selection of theoretical and re- 
search studies and a somewhat larger collection 
of "how-to-do-it," or technical assistance mate- 
ria^5 such as guides and manuals. 

In order to give the reader the flavor of what 
has been written on citizen evaluation and moni- 
toring as a backdrop for the Casebook, the fol- 
lowing two-part review of materials is presented. 
The first section describes the evolution of citi- 
zen evaluation and monitoring in generic services 
and in mental disabilities, and the second section 
briefly summarizes the materials that are avail- 
able to citizens to assist them in implementing 
evaluation or monitoring strategies. 



Evolution of Otizen Evaluation ana 
Monitoring 



Citizen Participation in Social and 
Economic Programs 

The anteceaents of citizen evaluation and mon- 
itoring can be found in the citizen participation 
movement that began in the 1960s and included 
such concep'vS as "maximum feasible participation 
of the poor" and "comm'jnity control." The ini- 
tiation of the Great Society programs and their 
concomitant requirements for citizen partici- 
pation spurred the development of nimierous ar- 
ticles and books that describe the potential and 
limitations of citizen involvement in neighborhood 
development, urban renewal, health planning, ed- 
ucation, and other areas (Brody 1970; Marsliall 
1977; Piven 1966; Slaver 1970). 

Community participation i.i federally funded 
programs was lae forerunner of other related 
citizen review activities including advocacy re- 
search, consumen'^m, and the self-help move- 
ment. Various studies concerning either citizen 
participation or its off-shoots, such as consumer- 
ism, describe the level of influence or power that 
citizens can obtain, the ways in which citizen 
participation can be effective, and the limitations 
of increased citizen participation in decision- 
making. Certain authonr have suggested various 
classifications, such as the citizen participation 
ladder that arrays citizen boards on a continuimi 
from purely advisory at one end to citizen control 
at the other end (Amstein 1969). Others have de- 
veloped strategies of citizen participation in 
eluding staff supplement, cooptation, eidvocacy, 
and community power (Burke 1979). Many studies, 
however, simply describe and compare the use of 
citizen boards or other participatory mechanisms 
to influence public policy (Mogulof 1969; Vanecko 
1969). 

Citizen Participation in Mental Health 

As one observer notes, "while there have been 
numerous studies of the participation of the poor 
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in a wide variety of programs, citizen partici^ 
pa'cion in mental health programs has not gener- 
ally received the same degree of attention" 
(McCord 1982). This same author goes m to sug- 
gest that this disparity in the literature should not 
exist since mental health associations and state 
hospital boards of visitors involve a tradition of 
citizen participation in mental health programs 
that substantially predates recent developments 
in other programs. 

Some studies in this area highlight the early 
barriers to citizen involvement 5n the CMHC 
program. For instance, an analysis reported in 
1974 notes the lack of emphasis on citizen par- 
ticipation in the early stages of the CMHC pro- 
gram, especially the lack of legal authority of 
citizen advisory boards (Chu and Trotter 1974). 
Another observer cites the initial difficulty of 
defining "community*' and, consequently, the role 
of citizens and consumers in plsmning and oper- 
ating community mental health services (National 
Institute of Mental Health, undated). 

Several studies that exan^ine citizen partici- 
pation in mental health have focused on the 
problems encountered by volunteers, especially 
lack of power and effectiveness in their roles. For 
example, a 1974 study of the perceptions of com- 
munity mental health center board members in 
the State of Tennessee revealed that half the 
respondents thought that their actual power to 
influence was quite limited (Robins and Blackburn 
1974). Other studies have identified problems 
common to citizen boards including role con- 
fusion, lack of staff support, and lack of respon- 
sibility for decisionmaWng (Morrison et al. 1978). 
At the same time, certain observers noted that 
citizen participation in mental health services 
must be tempered by political realities and 
adapted to express community needs and goals 
(Katie 1975). 

More recently, a study of mental health center 
boards in New York and Pennsylvania examined 
the division of responsibilities between boards and 
program directors. The author suggests that de- 
spite a lack of clear, formal differentiation of 
responsibilities between boards and program di- 
rectors, there was little conflict since boards do 
not seek to expand their influence but generally 
defet responsibility for most tasks and decisions 
to agency directors (McCord 1982). In another 
study of area mental health boards in Massa- 
chuse'cs, the authors found that four tjrpes of 
board accomplishment- service creation and im- 
provement, outside resource mobilization, local 
autonomy, and coordination— were positively in- 
fluenced by organizational characteristics and 
attitudinal variables (Dorwart and Meyers 1981). 

Despite the limitations of citizen participation 
in mental health programs, the importance and 
value of such input has been reaffirmed in most 
studies, and as suggested in the Massachusetts 
area board study, citizen participation has been 
extended to include new dimensions such as re- 



view and evaluation of mental health center 
programs. 

Development of Citizen Evaluation 
and Monitoring in Generic Services 

It is not surprising that citizen participation 
expanded into tiie arezis of monitoring and eval- 
uation of publicly financed programs. A major 
impetus for this was tlxe decentralization of many 
Federal gr?nts programs (e.g.. General Revenue 
Sharing, Community Development Block Grants, 
Comprehensive Employment Training Assistance, 
etc.) and the Federal requirements that citizens 
should be involved in the evaluation of local pro- 
grams. Citizen evaluation and monitoring has also 
encompassed nursing homes, schools, biomedical 
developments, and environm'^ncal issues. 

The expansion of citizen evaluation and moni- 
toring in such disparate areas ais community de- 
velopment and biomedical innovation wais in- 
fluenced by the need for greater public account- 
ability in these complex programs. Equally im- 
portant was a general dissatisfaction with the 
usefulness of evaluation research and the absence, 
very often, of those persons most affected by the 
program in the evaluation activity. With respect 
to the latter criticism. Federal agencies such as 
the National Institute of Education introduced the 
"stakeholder" approach to evaluation in an effort 
to include a broad spectrum of users and con- 
sumers in the evaluation activity. More recently, 
researchers have criticized this approach as it 
was applied in certain federally funded evalua- 
tions (Bryk 1983). 

Citizens have taken a more proactive role in 
evaluating local school programs. The National 
Committee for Citizens 5n Education (NCCE), a 
nonprofit public interest organization, has pre- 
pared numerous documents to assist parents and 
others to evaluate local schools. For example, the 
author of one document describes the charac- 
teristics of what is generally believed to be a 
good school and the ways that citizens can es- 
tablish their own standards for determining the 
quality of the education their children receive 
(NCCE 1982). 

The importance of community participation in 
evaluation activities is discussed by several ob- 
servers within the context of a successful com- 
munity development effort in a rural Missouri 
community. The authors of this study suggest that 
participatory evaluation can increase the com- 
munity's knowledge of itself, what it accom- 
plished and what it failed to do. Moreover, the 
results of the evaluation reflect those areas in 
which the community has the greatest interest 
and are presented in lay terms, not professional 
jargon (Lackey et al. 1981). 

Citizen participation in community develop- 
ment decisions would seem to be a logical role, 
given the publicly oriented focus of the program. 
Such involvement, however, in highly technical 
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and scientific areas is not as easily accepted. One 
observer suggests that the general public has the 
ability to participate in complex technical areas 
and discusses those efforts in the context of four 
controversial medical innovations: DES (the 
morning-after pill), the artificial heart program, 
the swine flu immunization program, and recom-^ 
binant DNA research. As noted by the author, 
three themes concerning the way in which lay 
citizens influenced these controversial areas re- 
cur in all four efforts: 

. . . reasc'^'Jsing risks and benefits; broadening 
and humanizing the perspective taken on is- 
sues; and trying to assure sound and legiti- 
mate decisions \.>*rcugh a more democratic 
decision process (Dutton 1982). 

Citizen monitoring examples are also prevalent 
in generic services. As noted earlier, decentral- 
ization of Feder?! grants programs spawned sev- 
eral national citizen monitoring efforts that were 
organized around local grassroots coalitions. As 
one observer notes, perhaps the most comprc 
hensive citizen monitoring project to date has 
been the assessment of Community Development 
Block Grants (CDBG) conducted by a coalition 
called the Working Group for Community Devel- 
opment Reform (Eisenberg 1981). Financed 
through a Title IX grant from the Community 
Services Administration, national staff developed 
a research design and a set of survey forms that 
were used by all participating local groups to 
monitor and evaluate the CDBG program. Since 
the monitoring project began, two major docu- 
ments have been produced that demonstrate the 
significance of citizen monitoring as a local and 
national resource, as well as the inadequacies of 
the CDBG program (Center for Community 
Change 1980). 

Citizen monit nng efforts m children's serv- 
ices, nursing homes and Federal programs (Medi- 
caid, Title XX, etc.) have ^^so 'jeen documented. 
For example, since 1974, tne Community Council 
of New York has conducted more than 15 moni- 
toring projects of publicly funded social service 
programs and experimented with a veuiety of ap- 
proaches, including the use of jotimalism students 
to monitor a new income maintenance program 
(Community Council of Greater New York 1979). 
In another example, the Massachusetts Office for 
CMldren established an institutional review com- 
mittee that grew out of its state>Adde network for 
citizen participation. As stated by the authors of 
a handbook on citizen review, 

institutional review by citizens is not just 
another agency evaluation or licensing visit 
. . . [it's] people who believe in their com- 
munity's responsibility for its children 
(Goldman et al. 1980). 

And finally, increasing public concern regarding 
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the quality of care provided to nursing home 
residents has prompted several documents ex- 
ploring the use of volunteers to evaluate or mon- 
itor nursing homes (Durman et al. 1979). 

Much of what has been written on citizen 
evaluation or monitoring describes traditional, 
and fairly straightforward, approaches. Certain 
system observers, however, would include non- 
traditional approaches such as legal and jour- 
nalistic models (investigative reporting) among 
those methods that could be used to evaluate or 
monitor public activity. According to these 
authors, legal and journalistic approaches ema- 
nate from a hvimanistic rather than a "rational 
service tradition," and both are case oriented 
(Levine 1982). This same system observer would 
also include whistleblowers and individual ef- 
forts such as ly'^ Gibb's investigation of the 
health effects of toxic chemicals in Love Canal 
under the heading of citizen evaluation and mon- 
itoring. 

In all of the approaches described in the lit- 
erature — both traditional and nontraditional — a 
major theme that emerres is the difference in 
approach between lay or volurteer evaluators/ 
monitors and professionally directed program 
reviews. Observers in the mental health field have 
looked at these differences in formulating a 
conceptualization of citizen evaluation and 
monitoring. 

Citizen Evaluation and Monitoring 
in Mental Health Services 

Since the enactment of the 1975 Amendments 
to the Community Mental Health Centers Act 
(Public Law 94-63) requiring citizen review of 
center services, several reports have been pre- 
pared that examine the implementation of that 
requirement, as well as general citizen involve- 
ment in mental health evaluation and monitoring. 
As noted by some authors, the significant dimen- 
sion of citizen evaluation is the participants* roles 
and concomitantly the amount of irifluence or 
power they can exert in those roles. Similar to the 
classifications developed in the late 1960s with 
respect to citizen participation, several re- 
searchers have developed a classification of cit- 
izen roles in evaluation — ranging from the most 
passive (i.e., subjects of evaluation and recipients 
of information) to the most active (i.e., partners 
in evaluation or independent evaluators) (Dinkel 
et al. 1982). These roles are then analyzed by the 
tjrpe of potential participant involved (consumer, 
community, program employee) and the specific 
evaluation functions they might perform. 

One role in particular, the solicited reactors to 
evaluation, was mandated by the 1975 Amend- 
ments to the CMHC Act and led to further re- 
search and ultimately the development of a 
mechanism taiown as the citizen review group 
(CRG). The CRG process varies in who assumes 
primary responsibility (evaluation staff, agency 
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board, or external citizen group) and is a rela- 
tively undemanding form of citizen participation 
in evaluation (Zinober et al. 1980). 

These and other studies reveal that for the 
most part CMHCs have not sought out the pub- 
lic—other than their own boards — in evaluation 
activities (Flahe-^ and Olsen 1982). Even CMHC 
boards, accordiiig to some observers, have shied 
away from active participation in evaluation ac- 
tivities* In a recent study of center boards in New 
York and Pennsylvania (McCord 1982a), three of 
the five boards analyzed indicated that agenc:^ 
review and evaluation was primarily a respon- 
sibility of the agency director. As noted by the 
author, board members were more willing to as- 
sume or accept responsibility in areas that ap- 
peared less "technical," such as where to locate 
agency facilities. In general, boards are uncertain 
as to how to ev2duate a program's performance, 
or what criteria or standards to utilize — questions 
that plague prof essional evaluators as well 
(McCord 1982b), 

One important form of evaluative information 
is consumers* satisfaction. In a review of existing 
studies, Lebow (1982) observed that assesspient of 
consumer satisfaction with mental health services 
was unusual IS years ago, but today such surveys 
are standard parts of the practice of many mental 
health facilities, especially community mental 
health centers. It should be noted, however, that 
since this type of evaluation is no longer required 
of CMHCs, its importance may diminish. Since 
there is a large but unorganized body of literature 
OP. tWs subject, readers who are interested in 
pursuing consumer satisfaction in more detail 
should first explore the literature reviews that 
are available. Although certain drawbacks to 
consumer evaluation have been documented, some 
evaluators have stressed its importance in making 
programs more responsive and, in effect, pro- 
tecting consumers from potential problem situ- 
ations (Quilitch and Szczepaniak 1976). 

Recently, consumer evaluation has involved 
nontraditional subjects such as children and their 
families participating in treatment planitog con- 
ferences and the review of such conferences 
(Favaro and Love 1983). Parents often differ from 
professionals in what they expect from services 
and evaluate services from a different vantage 
point (Sommers and NYCA 1979). 

Some of the major differences in lay and pro- 
fessional evaluations have been described by 
various system observers. For example, authors 
have suggested that findings from citizen evalu- 
ations are more useful because (1) citizens have 
greater access than professional evaluators do to 
implementation strategies (e*g., contacts with the 
le^lature axid the media) and (2) the conununity 
ultimately Inherits the results of the evaluation, 
rather than the professional e valuator or program 
administrators vAio may be personally less com- 
mitted to certain types of program cbanges 
(Windle 1976). Other differences include the lay 



evaluator's use of conimon sense rather than 
theories for guidance and values that counter the 
elitist domination of professional evaluation and 
democraticize program administration. In addi- 
tion, using citizens to perform certain evaluation 
tasks leaves the professional evaluators free to 
focus on those areas where their skills and con- 
tributions are unique. Finally, citizen evaluators 
car also help public agencies save limited funds 
(Windle 1976). 

There are, howe/er, drawbacks to using citizen 
evaluators. As noted earlier, many of the limi- 
tations of lay evaluation or monitoring can be 
attributed to the role that citizens play in such 
activities . For example, some observers have 
found that lay citizen groups seldom function as 
completely independent evaluators but are most 
likely to evaluate in cooperation with agency 
staff (Zinober et al. in press). Thus, citizens out- 
side of the agency being evaluated are often 
limited in what they can learn. A second draw- 
back may be that many lay citizens have a strong 
interest in service expansion since they will be 
either directly or indirectly affected by the re- 
sults of the evaluation. Moreover, the extent to 
which lay evaluators actually represent the larger 
community, as in ot'*:er citizen-directed activ- 
ities, remains a probi...n. Also, citizens, like pro- 
fessional evaluators, will bring their own biases to 
the evaluation; however, such biases can be min- 
imized through training and exposure to a broad 
range of ideas (Hessler and Walters 1976). 

This last point does suggest that in some re- 
spects citizen evaluators are similar to profes- 
sionals. Perhaps what is most unique about citizen 
evaluation or .monitoring is that it provides an 
outsider's view of the system. What is often 
lacking for many potential citizen evaluators or 
monitors is the necessary tools, training, and ex- 
posure to evaluation methods. The final section of 
this literature review briefly describes what is 
available to assist consumers and other citizens in 
the performance of evaluation and monitoring. 

Technical Assistance for Citizen 
Evaluation and Monitoring 

Traditionally, materials on program evaluation 
have been relatively technical, but recently a 
number of manuals and monographs on mental 
health evaluation in general and citizen involve- 
ment in evaluation in particular have been 
published. 

In A Trust of Evaluation: A Guide for Involving 
Citizens in Community Mental Health Center 
Evaluation (Zinober and Dinkel 1981), the auth- 
ors describe a project in which 17 CRGs were set 
up to review the evaluation reports of seven com- 
munity mental health centers. This monograph is 
useful for citizens desiring to become involved in 
evaluation. It covers such areas as CRG member- 
ship, recruiting, group dynamics, and the flow of 



19 



information between the center and the citizen 
group. 

A second monograph. Citizen Roles in Com- 
TTumity Mental Health Center Evaluation: A 
Guide for Citizens (Peters at al. 1979), describes 
the way in which citizens can use evaluation to 
regain some control over agencies that serve their 
communities. The monograph starts by explaining 
the evaluation process in laymen's terms and de- 
scribes a variety of activities that can be per- 
formed by citizen groups, including preparing in- 
formation about the center, reviewing the cen- 
ter's plans for evaluation, and conducting simple 
evaluation studies. Examples of relatively simple 
studies such as surveys of consumer satisfaction 
and awareness and attittades of community groups 
toward the center are presented. 

A third resource for citizens is Citizen Evalu- 
ation of Mental Health Services: An Action Ap- 
proach to Accountability (MacMurray et al. 1976). 
Like the materials described above, this publica- 
tion focuses on accountability and describes this 
process in four steps: (1) organizing the citizen 
accountability team; (2) assessing mental health 
problems and resources; (3) evaluating specific 
programs and services; and (4) using the results of 
those evaluations. The authors describe various 
a* "poaches to community needs ?*ssessment and 
r )vide additional advice to citizens on what to 
evaluate and how to collect data. Finally, another 
useful resource for citizens is the Mental Health 
Association's Community Mental Health Center 
Site VisitaHon Handbook (1976). This handbook 
includes a site visitation form and the types of 
information that are necessary in reviewing cen- 
ter services. 

In addition to the publications that deal directly 
with the issue of citizen involvement in mental 
health evaluation, a r^imber of recent publications 
address mental health evaluation in general. 

Though directed toward administrators and 
professional evaluators. Resource Materials for 
Community Mental Health Program Evaluation 
(Hargreaves et al. 1979) is a useful resource for 



citizens since it present*, both background infor- 
mation on evaluation and case examples in clear 
and relatively nontechnical terms. Also of use to 
lay citizens are those articles in the document 
that address evaluation of service effectiveness. 
This particular section contains a review of 
measurement instruments and references for 
further information. 

. Another useful resource is Evaluation in Prac- 
tice: A Sourcebook of Program Evaluation Studies 
for Mental Health Care Sys^^ms in the United 
States (Landsberg et al. 1979). The editors of this 
monograph have compiled a series of cases that 
illustrate program evaluation as it is actually 
conducted in mental health faclities. Again, the 
intended audience is program administrators and 
evaluators; however, the cases are understandable 
and highly instructive for laymen as well as for 
professionals. Topics such as program accept- 
ability, availabilit;,', accessibility, and awareness 
are illustrated in the cases and should be par- 
ticularly useful for citizens who are interested in 
examining these issues in their communities. 

More ".echnical information on methods and 
research designs is presented in a monograph en- 
titled A Working Manual of Simple Program 
Evaluation Techniques for Community Mental 
Health Centers (Hagedom et al. 1976). Although 
this manual presents more sophisticated meth- 
odological information than the other publications 
cited here, it might be helpful for those citizens 
who are undertaking more ambitious projects. 
Methods for needs assessment, analysis of pat- 
terns of service use, cost analysis, and studies of 
client outcome are described in the manual. 

The remaining chapters of the Casebook de- 
scribe the 51 cases selected. Each chapter begins 
with a boxed hypothetical case that illustrates 
some of the major issues discussed in the chapter. 
This is followed by a chapter introduction that 
summarizes the actual cases. Finally, each chap- 
ter includes a number of cases that have oeen 
selected because of their relevance to the cliap- 
ter's concerns. 
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Where Are You in the System? 



Doug Brown is an active member of the Greene County Mental 
Health Association (GCMHA). The association, composed of interested lay 
persons, consumers, and mental health professionals from the conmiunity, 
has recently become concerned about the need for short- and long-range 
planning in the county's mental health system. The need for planning has 
been underscored by recent funding cutbacks and pressure to reexanaine the 
county's service priorities. In order to prepare a plan, however, there is a 
need for an evaluation of the county's mental health needs and service 
availability. In response to this concern, the board of directors of the 
GCMHA set tip a planning and evaluation conunittee and appointed Doug 
Brown as chairman. 

Doug and his committee have a number of key decifdor*s to make be- 
fore actually collecting information and developing planning and policy 
reccmmendations. The first consideration that Doug must put before the 
committee is a discussion of the inherent strengths and limitations of any 
GCMHA-sponsored evaluation oased on the organization's position in the 
system and its reputation in the county. One of tlie strengths that Doug can 
count on is the association's independence and freedom from any potential 
conflict of interest. Further, the association has worked hard to reflect the 
interests of all elements of the community in its advocacy for improved 
services and has therefore won the respect of the major constituencies in 
the county. Finally, GCMHA has been very active in the State capital and 
has successfully lobbied for increased funding for the Greene County Men- 
tal Health Department. As a result, GCMHA has a cooperative and rea- 
sonably productive relationship with the county mental health agency. 

Unlike the comty's mental health advisory board, however, Doug and 
his conunittee do not have automatic access to county-supported services 
or to their records. Further, many of Doug's conmiittee members are not 
familiar wth the range of services offered in the county since GCMHA's 
focus for the last several years has been on children's programs. Finally, 
the evaluation is only one of GCMHA's activities, and therefore Doug and 
his committee will have to compete with other association conunittees for 
resources and attention. 

All of these considerations are crucial because of their influence 
upon the level of participation that Doug and his committee commit to the 
evaluation, the scope of their exploration, the issue or issues they take on, 
the means they choose to get answers, and the steps they take to ensure 
that their recommendations are integrated into county mental health pol- 
icies and plans. 



The cases that follow show citizens and con- 
sumers functioning in a variety of contexts from 
State advisory councils and agency boards to 
client groups. Each organizational loc^^^on brings 
with it a different perspective and orientation 
which in turn shapes the evaluation or monitoring 
activity conducted. 



State Council 

The first case in the chapter, written by Dennis 
Geersten and Jean Okawa, describes the activities 
of a State planning and advisory council that 
launched a statewide assessment of mental health 
problems and the needs of special populations 
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(e.g., chronically mentally ill persons). The broad 
mandate of the council coupled with its access to 
State-level planning processes means that it is 
capable of taking on more general and all- 
encompassing tasks and that its requests for in- 
formation are likely to receive swift and serious 
responses. Ftirther, given its position in the sys- 
tem, the coimcil's recommendations— as shown in 
the case summary — stand a good chance of being 
integrated into the State's mental health plan. 

Regional or County Board 

The next two cases involve sub-State citizens* 
groups, one on a regional level and the other on a 
county level. The regional advisory board, de- 
scribed by Jessica Wolf in case #3, has very spe- 
cific State-mandated responsibilities, and the 
format f or agency review and evaluation is 
equally specific. This particular regional citizens' 
group functions in tandem with local catchment 
area councils and works in partnership with the 
regional mental health commissioner. Because the 
regional body is responsible for funding recom- 
mendations, its reviews carry a great deal of 
weigjit. The formal power of the group also tends 
to facilitate the implementaton of recommen- 
dations. In this case, however, the formal position 
of the group may also have engendered a certain 
amount of defensiveness in the agency being 
evaluated given the fear of cutbacks in funding. 
The county advisory board, described by Dan 
Hustedt in case #2, also has the advantage of 
working undei a very specific State mandate that 
spells out the board's responsibilities for program 
review. The specificity and formality of its 
functions epjse the board's access to county- 
funded agoncies and guarantee at least an au- 
dience for its recommendations. The role and 
functions of both of these boards, however, dic- 
tate that the majority of their activities are 
carried out in partnership with mental health 
agency staff. 

Agency Board 

The fourth case, written by Jarrett Richardson, 
involves the activities of an agency board of di- 
rectors. The assessment described was motivated 
by the board's desire to assess agency perform- 
ance after 2 years of operation. The board, which 
is broadly representative of the community, has 
responsibilities in a variety of areas including 
personnel, admissions, finances, administration, 
and evaluation. The case illustrates that boards of 
directors don't have to confine their efforts to 
internal agency reviews, but can also use know- 
ledgeable persons in the community and agency 
consumers as sources of information. The case 
also raises the question of how much time should 
be devoted to evaluation tasks by board volun- 
teers with multiple and competing agency 
responsibilities. 



External Group 

Citizens' groups that function independently of 
a service system can also play a significant role in 
program monitoring and evaluation. Darlene 
Humphrey describes the experience of one such 
group whose mission is to monitor services for 
frail and vulnerable persons in m^rsing homes. The 
experience of Ms. Humphrey's group, like that of 
other monitors, reinforces the fact that the rou- 
tine presence of an outside observer can reduce 
abuse and neglect in residential facilities. The 
case also points out the profotmd influence that 
an external public interest group can have on a 
program— in this instance, the agency was suc- 
cessful in eventually prer<^uing the State to close 
a substandard nursing home. A disadvantage of 
being an external group is also pointed out by this 
case — the difficulty in securing access to facil- 
ities in order to carry out adequate monitoring. 

Consumer Groi^> 

Case #6 reflects the tjrpes of evaluation ac- 
tivities that can be undertaken by consumer or- 
ganizations—in this instance, a local mental 
health association. The case, written by i^ancy 
Sohlberg, shows how a local consumer group can 
both evaluate a major service provider and main- 
tain a mutually respectful and increasingly coop- 
erative relationship. The case further illustrates 
how a consumer groi:^ like a mental health as- 
sociation can sharpen its mission by including 
consumers of services as sources of information. 
In contrast to county and regional advisory 
boards, this case shows a possible strength as- 
sociated with independent consumer evalua- 
tors — the absence of defensiveness on the part of 
the subject of evaluation. 

Community MenUl Health Center Board 

In addition to the earlier case example of an 
agency board evaluation, the next case exem- 
plifies the sort of monitoring and planning ac- 
tivities that can be undertaken by a board of a 
community mental health center. The case, de- 
scribed by Roger Strauss, traces the evolution of 
a CMHC board's involvement from a somewhat 
passive role as recipient of information to one of 
active particiption in the assessment of needs and 
the development of rational funding strategies. 
The discussion also points out that the level of 
satisfaction of CMHC board members increases 
substantially as they increase their involvement. 
Finally, the case shows how CMHC boards can 
create new roles and responsibilities in program 
evaluation now that the Federal guidelines and 
site visits have been eliminated. 

Board of Visitors/Quality Monitors 

Case #8 revolves around the activities of a 
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board of visitors at a State hospital. It is reflec- 
tive of the activities carried out by such bodies 
and the tj^es of problems they are likely to en- 
counter. This case, written by Marilyn Seide, in- 
dicates that boards of visitors shouid sharpen 
their client- centered missions in order to avoid 
being drawn into staff-related concerns. It also 
points out the delicate balance that such groups 
of monitors must maintain between cooptation on 
the one hand and unproductive confrontation on 
the other. 

Advocacy Group 

The use of consumers of services as quality 
assurance monitors is the focus of case #9, re~ 
lated by David Schott, The organization spon- 
soring the monitoring activity is an advocacy 
group that represents the rights and interests of 
mentally disabled persons. Tne case shows how an 
advocacy organization can mobilize its clients to 



work for the improvement of services on their 
own behalf. It also suggests that advocacy groups, 
in particular, can be influential in advancing the 
consumer's view of service quality and contrasts 
this view with that of case managers and others 
vested with quality-assurance responsibilities. 

Client Group 

The final case in this series focuses on a patient 
group within an inpatient facility and discusses 
how such groups czn influence facility policy and 
assist in the resolution of client grievances. The 
case, written by Gabriel Manasse in consultation 
with Charles Gold, describes the evolution of the 
patient group and its increasing influence and 
range of activities. It also points out the diffi- 
culties of maintaining contintdty within such 
groups and the problems some clients may have in 
sitting in judgment on clinicians from whom they 
may also be receiving treatment. 



1. State Council Assesses Utah's Mental Health Needs 

Dennis C. Geertsen 

Utah State Division of Mental Health 

Jean Okawa 

Utah State Planning and AtMsory Council 



SUMMARY 

The Utah State Planning and Advisory Council, 
in collaboration with the Division of Mental 
Health, conducted a needs survey of key inform- 
ants throughout the State of Utah. Respondents 
indicated their priority selections of target 
populations whose mental health needs were not 
being met, and the major mental health problems 
and service needs associated with each target 
population. The evaluation results were initially 
shared with selected staff and advisory council 
members by means of an oral report and slide 
presentation. A written report is scheduled for 
completion. 

TYPE OF ORGANIZATION 

The advisory council is composed of 20 
residents from nine community mental health 
catchment areas throughout the State. Council 
members include housewives and working women 
(55 percent), educators (10 percent), lawyers (5 
percent), paraprofessionals working within the 
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state mental health system (10 percent), and 
mental health professionals within (25 percent) 
and outside (15 percent) the system. These people 
are appointed to the council by the State board of 
mental health. The role of the advisory council is 
to advise the Division of Mental Health in the 
development, coordination, and evaluation of 
comprehensive mental health plans for the state. 

EVALUATORS OR MONITORS 

This study was carried out by the Advisory 
Council and the Division of Mental Health 
evaluate! who acted as project coordinator. 



REASONS FOR EVALUATION 
OR MONITORING 

The combined views of key informants (knowl- 
edgeable persons) who are and who are not part of 
the mental health delivery system is an important 
component in assessing the mental health needs of 
the state. This study was intended to assess the 
perspectives of key leaders and non-mental health 
providers regarding priority target populations, 
problems, and service needs. The study was 
undertaken to gather relevant data for mental 
health planning in Utah. 
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LEVEL OF PARTfCIPATIQN 

Citizens were involved as both research 
assistants and key informants* As researchers, 
members of the advisory council assisted in 
planning the study, formulating questions, 
developing and pilot testing the questionnaire, 
developing the method for sampling potential 
study participants, and coordinating and encour- 
aging responG;?nts within their district to 
participate. Technical aspects of these tasks were 
carried out by, or under the supervision of, the 
authors. In addition to being involved as re- 
searchers, council members at State and local 
levels were designated as part of the sample to 
receive mail questionnaires. 

TARGET OF EVALUATION OR MONITORING 

This study focused on the multiple needs of 
persons residing in all service areas in the State 
of Utah. Four of the nine service areas lie 
exclusively in rural areas of the State, while most 
of the others have at least one county which is 
ruraL Seven of the areas are served by compre- 
hensive mental hbalth centers. There is only one 
State hospital, which underscores the strong 
emphasis in Utah on community-based care. Utah 
is predominantly white (92 percent) and has a 
relatively large household size, averaging 3.2. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Issues evaluated in the needs assessment 
included target populations and problem areas felt 
to be of interest to public mental health programs 
in Utah. The target populations were: (1) ethnic- 
racial minorities, (2) chronically mentally ill, (3) 
elderly, (4) adolescents, (5) children, and (6) the 
general population. Rather than evaluating 
service programs, this study focused on specific 
problems related to mental dysfunction. 

TECHNIQUES USED 

The project coordinator and the chairperson of 
the advisory council designed the key informant 
survey, drafted the questionnaire, selected the 
random sample, mailed out questionnaires, and 
analyzed and reported the results. The group of 
respondents included all local advisory members 
and a randomly selected sample of other names 
(up to 50 persons) from lists that were identified 
as key referral sources. The total sample was 185 
persons, of whom 45 percent sent back 
questionnaires* 

FINDINGS OF EVALUATION OR MONITORING 

Preliminary results for the State as a whole 
include the ranking of mental health problems for 
six target populations. The ranking of target 



populations whose needs are perceived as not 
being met were also -ixaminel The three highest 
ranking problems were ••disturbed family rela- 
tions,- -alcohol abuse,- and -social isolation.- 
However, these and other problems varied 
somewhat by target population. Target popu- 
lations not being adequately served were ranked 
in this order: children, adolescents, and elderly 
(tied for first); ethnic/racial minorities (second); 
the chronically mentally ill (third); and the 
general population (last). Former Federal 
mandates and current State mandates specify the 
chronically mentally ill as a high priority. 
Respondents to this study apparently felt that ' he 
needs of this population were being met, at least 
in relation to other target populations. 

RECOMMENDATIONS 

Detailed recommendations will be developed 
upon completion of the analysis and report. Two 
general recommendations are as follows: (1) State 
and Federal mental health agencies should 
reconcile differences between need priorities of 
target populations with those priorities identified 
by other local referring agencies, and (2) citizen 
participation should more often be included in 
future evaluations, recognizing that this in- 
volvement can be cost effective and contribu'^v^ to 
study quality. 

STEPS TO ENSURE IMPLEMENTATION 

The following implementation steps were 
projected: (1) present preliminary results to the 
advisory council, the division of mental health, 
local and State boards, and program directors; (2) 
include results in early drafts of the mental 
health plan; (3) develop detailed recommendations 
and a final report; (4) disseminate the final report 
to the above groups and other interested agencies; 
and (5) assess utilization of study results through 
followup contacts with the responsible agencies. 

EXTENT OF IMPLEMt^NTATIQN 

Preliminary reports have been presented to the 
advisory council, the division of mental health, 
and the mental health plan steering committee. A 
written summary has been included in a draft of 
the State mental health plan. Other steps are now 
in the process of implementation. 

SPECIAL BARRIERS OR SUPPORTS 

Because of limited resources and small staff 
(one evaluator). the study could not have been 
undertaken without citizen involvement. The 
advisory council members were viewed by the 
State Division of Mental Health as being experts 
in their local geographic areas pertaining to 
sample selection, coordination, and identification 
of problems that subsequently became content for 
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the questionnaire. Council members helped to 
legitimize the study by making contact with other 
key informants in their own local communities. 

Some nouncil members felt that the project 
moveo too slowly in the early phase. This 
situation was due to the limited research 
experience of council members and unavailability 
of technical assistance to the council. The lack of 
concentrated time and effort by advisory council 
members (i.e., infrequent monthly meetings) was 
an additional barrier. However, the above 
problems were resolved and the project proceeded 
once the technical research consultation was 
made available by the State Division of Mental 
Health. Research and financial aid from the 
division were essential resources in planning, 
implementing, and completing the study. 

RESOURCES AND COSTS 

The total cost for the study was $8,368. 
Approximately 68 workdays were used to carry 
out the project, including 40 days by council 



members (cost: $2,268) and 27 days by State staff 
(cost: $3,350). These personnel costs were donated 
to the project by the State agency and individual 
council members. Additional expenditures for 
travel ($2,500) and printing ($250) resulted in 
total out-of-pocket expenses of $2,750. Almost 
three-fourths of the total cost of the study was 
donated by agency and council volunteers, who 
proved to be important resources. 



ADDITIONAL COMMENTS 

Study participants felt that this project was a 
very successful example of a collaborative 
research endeavor between State and local 
agencies and citizens. A major benefit of the 
project was the inclusion of a broad-based 
perspective of mental health needs for State 
planning. Another benefit is the research resource 
provided by active volunteer commnnity parti- 
cipants, which is especially an asset during times 
of tight agency budgets. 



2. Site Visit Reviews in Monterey County 

Dan Hustedt 

Monterey County Mental Health Advisory Board 



SUMMARY 

The Monterey County Mental Health Advisory 
Board (MHAB) conducts an annual site review of 
each program receiving fun*^ ng under California's 
mental health statute— the Short-Doyle Act. 
Board members examine service goals and 
objectives, types of services provided, primary 
problems dealt with, existing alternatives for 
service, facility operations, location, service 
capacityr utilization, staffing, program organi- 
zation, and administration, unmet needs for 
service, and referral patterns. Site review reports 
which include MHAB conclusions and recom- 
mendations are written and distributed to board 
members and service providers. Service providers 
are asked to respond in writing to conclusions and 
recommendations of the site review team. 

TYPE OF ORGANIZATION 

The membership of the Monterey County MHAB 
and county boards throughout California is 
mandated by legislation that also governs 
community mental health programs. The board 
must include 17 members — 5 must be represen- 
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tative of selected disciplines, 2 must be physi- 
cians, 5 must either be consumers or have 
immediate family members who have been 
consumers of mental health services, and 4 
members must be representatives of the public 
health interest in nental health. In addition, by 
law the board must be representative of the 
county population in terms of age, sex, and race 
or ethnic group. The duties and functions of the 
MHAB are as follows: 

• Review and evaluate the community's mental 
health needs, services, facilities, and special 
problems 

• Review the county mental health plan 

• Approve the mental health planning process 

• Act in an advisory capacity to the local 
mental health director and the board of 
supervisors 

EVALUATORS OR MONITORS 

Each site review team included the local 
mental health director, the program evaluator, 
clinical personnel with some day-to-day dealings 
with the program being reviewed, and at least two 
representatives of the MHAB. 
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REASONS FOR EVALUATION OR MONITORING 

The annual site reviews are an integral part of 
the Short-Doyle planning process. The 
information from the reviews provides board 
members with some of the knowledge they need 
to make decisions concerning planning for 
program increases or reductions. It also provides 
an opportunity for clinicians in the system to 
become more knowledgeable about other 
programs and to '.supply constructive criticism as 
needed. 

LEVEL OF PARTICIPATION 

MHAB members participate in the following 
ways: 

• Review and approve the site review protocol 

• Attend the site reviews 

• Provide ideas for the conclusions and 
recommendation section 

• Review and approve both the consensus site 
review writeups and the service provider 
feedback reports 

TARGET OF EVALUATION OR MONITORING 

The site review process was repeated at each of 
the 20 mental health programs funder" by 
Monterey County. Operating under State legis- 
lation mandating community mental health 
programs for counties with populations over 
100,000, the Monterey County mental health 
budget for 1982 is slightly less than $7 million. 
The services are designed to serve a population of 
292,000. The full range of mental health services 
is available in both of the county's two catchment 
areas. In keeping with State prioritie3, over half 
of the services are operated through contracts 
with private agencies. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Board members examine service goals and 
objectives, types of services provided, primary 
problems dealt with, existing alternatives for 
service, facility operations, location, service, 
capacity, utilization, staffing, program organi- 
zation and administration, unmet needs for 
service and referral patterns. All services are 
rated In terms of the Importance of their goals 
and the program's efficiency in meeting these 
goals. 

TECHNIQUES USED 
For each service, an agreed upon set of 
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statistics depicting both the program activities 
for the previous year and relevant cost data Is 
supplied to all reviewers. The site review protocol 
Is then administered and site review conclusions 
are recorded. 

FINDINGS OF EVALUATION OR MONITORING 

A site review report for each program was 
distributed to all board members and to each 
service provider. Each service provider was asked 
to respond In writing to any questions, comments, 
or suggestions and criticisms recorded In the site 
review conclusion section. A document titled 
Provider Feedback to MHAB Site Review 
Conclusions was then prepared. 

In addition to the findings described In the site 
visit report, reviewers are also required to rank 
the goals and objectives of each program 
according to priority and to assess the progr^iTi's 
efficiency. This Information Is later used as Input 
to MHAB's consensus ratings of all mental health 
services offered In the county and the consensus 
ranking of all needed services. 

RECOMMENDATIONS 

Recommendations were included in the 
consensus writeups and responded to In the 
Provider Feedback document. In addition, the 
advisory board's priorities were used to assist In 
determining necessary program reductloi s for the 
following fiscal year. 

STEPS TO ENSURE IMPLEMENTATION 

The board asked that providers respond to their 
recommendations. The recommendations, 
however are not binding, and service providers 
are ultlnri >tely responsible to the county program 
direction In regard to operations. Since the 
reviews are made on an annual basis, service 
providers are called upon to address problems 
they had agreed to remedy from the previous 
year's review. In practice, many of the MHAB 
recommendations have been Implemented by the 
providers. 

EXTENT OF IMPLEMENTATION 

With major funding reductions In the mental 
health program In the past year, the county 
mental health program eliminated the lowest 
priority services as rated by the MHAB. 

SPECIAL BARRIERS OR SUPPORTS 

The program chief and the MHAB staff member 
arranged for the reviews and coordinated all the 
scheduling of clinical time for the review effort. 
All of the programs cooperated In providing the 
necessary information to make the review process 
effective. 
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RESO^jRCES AND COSTS 

The costs for mailing, typing, and xeroxing 
were absorbed by program administration. The 
other costs wore the approximately 70 hours of 
the program chief's and evaluator's time in 
attending the reviews* Also, the time of various 
clinical reviev^ team members and service 
providers was an additional cost to the rystem* 

A DDITIONAL COMMENTS 
The findings of the review teams ^re not always 



the most important element of the process. The 
concept behind the reviews is that they allow the 
MHAB members a means of regularly studying 
mental health system operations so that they can 
intelligently communicate the community's 
perspectives to the policymaking process. Without 
the annual reviews, board members do not feel 
adequately informed to evaluate the community's 
mental health needs, services, facilities, and 
special problems as they are mandated by law to 
do. 



3. Cohaborative Regional Review of Keystone House 

Jessica Wolf 

Southwest Regional Office 
Connecticut State Department of Mental Health 



SUMMARY 

Connecticut law mandates that regional mental 
health boards (RMHBs), subarea catchment area 
councils (CACs), and the State-employed regional 
mental health director regularly review programs 
funded with State department of Mental Health 
community grants-in-aid. According to the law, 
the regional mental health boards, together with 
the regional mental health director, must make 
recommendations to the commissioner of the 
department of mental health concerning the 
funding of community mental health programs. 

The subject of this review is Keystone House— 
an agency that operates a halfway house and an 
independent living apartment program. Findings 
of the review included the need to increase 
occupancy rates, change admission requirements, 
and develop measurable goals. Though there was 
some conflict between reviewers and the agency, 
the spirit of compromise prevailed and the 
recommendations were Implemented. 

TYPE OF ORGANIZATION 

The regional mental health board (RMHB) 
consists of 16 members of whom four are 
appointed from each of the four CACs. A 
majority of the members of the RMHB must be 
consumers (i.e., nonproviders of mental health 
care). RMHBs are funded through State depart- 
ment of mental health grant-in-aid funds as well 
as through local funds. The regional boards 
employ their own staff. 

RMHB procedures place the initial respon- 
sibility for program review at the CAC level. A 
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panel, generally composed of three CAC mem- 
bers, RMHB staff, regional director, and assistant 
regional director, meets after the staff has 
developed prelir 'nary information concerning the 
agency, program, and financing. The panel 
members determine the questions they wish to ask 
during the site visit. The site visit is held at the 
agency and includes representatives of staff and 
the agency board. Following the site visit, the 
panel reconvenes to discuss the visit and issues 
raised there. An RMHB staff person then develops 
a preliminary review report, which is circulated 
to the panel members, the remaining members of 
the CAC. and the agency. 

The next step is consideration by the CAC at a 
regular monthly meeting. Agency staff and board 
members are invited tc answer questions. After 
CAC members vote, the review report is circu- 
lated to the members of the RMHB, who discuss 
the report with agency representatives. The 
regional mental health director is present 
throughout all these meetings. After the final 
action is taken, the regional mental health 
director forwards the review report to the 
commissioner of mental health and agency 
representatives. 

EVALUATORS OR MONITORS 

In general, efforts are made to include both 
consumers and service providers on review panels. 
In the Keystone House review, there were two 
providers and one consumer. One provider was the 
administrator of a nearby drug abuse treatment 
program, and the second was the administrator of 
a private psychiatric hospital. The consumer was 
the administrator of a local transportation agency 
who had been active in mental health affairs and 
had at one time been a board member of Keystone 
House. The executive director of the regional 
mental health board, the regional mental health 
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director, and the assistant regional mental health 
director completed the team, 

REASONS FOR EVALUATION OR MONITORING 

As indicated above, State, law requires review 
' the RMHB and regional director, 'n Region I of 
necticut, full reviews of the 20 community 
grantee agencies are undertaken at least once 
every 3 years. Partial reviews may be undertaken 
when indicated. 

LEVEL OF PARTICIPATION 

Citizens serving on the review panel, CACs, 
and RMHB participated as partners or colleagues 
with RMHB and State staff in the assessment of 
Keystone. They were joint participants in the 
collection and assessment of infornr»ation and the 
formulation of recommendations. 

TARGET OF EVALUATION OR MONITORING 

The Keystone agency includes a halfway house 
and a supervised apartment program for persons 
discharged from psychiatric hospitals. The service 
area consists of four towns, although Keystone 
also takes referrals from other parts of the region 
if beds are available. Clients are referred 
primarily from a general hospital, the State 
hospital serving the region, and a private 
psychiatric hospital. The halfway house presently 
has ten beds; however Keystone has received an 
award from the Department of House and Urban 
Development (HUD) to renovate the house and to 
add two beds. The apartment program consisted 
of two rented apartments with a total of five 
beds, although it was realistically possible to use 
only four. The program is staffed by an executive 
director and the equivalent to 4 additional 
staff, primarily counselors. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The review focused on programmatic and fiscal 
issues including the role of the board of directors, 
staff assignments, the status of the HUD grant, 
interagency relationships, ease of referrals, and 
foUowup of former residents. Specific issues of 
concern to the panel included the costs of the 
apartment program, its apparently low occupancy 
in the previous year, and the agency's policy that 
only graduates of the halfway house could be 
admitted to the apartment program. In addition, 
there was concern that the agency was receiving 
funding for five beds when in fact only four could 
be used. 

TECHNIQUES USED 

A document titled Stcmdards for Funding was 
mailed in the fall of 1981 to the executive 
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director of the agency with a request for 
response. In addition, service utilization infor- 
mation was gathered by the RMHB and the 
regional director was responsible for collecting 
cost data. Panel meetings, a site visit, a meeting 
with board members and staff, and CAC and 
RMHB meetings, also occurred. Additional data 
were requested from the agency, including more 
up-to-date information on costs and plans for 
interrelating the HUD grant with ongoing agency 
operations. Dissatisfaction was expressed by the 
review panel because, according to panel 
rrembers, they had difficulty in obtaining 
complete information. Agency representatives 
complained that the information requested, once 
provided, was not considered carefully enough. 



POINDINGS OF EVALUATION 
OR MONITORING 

In the first draft of the review, the panel 
recommended defunding the apartment portion of 
the Keystone program because the occupancy was 
low (54 percent) for the previous year (1980); the 
State grant stipulated at least 80-percent 
occupancy; and the low occupancy rate resulted in 
higher unit costs. The draft report also noted the 
agency's inability to accept outside clients to the 
apartment program. At this stage, there was a 
climate of distrust and even hostility between 
members of the review panel and some agency 
representatives. Agency representatives com- 
plained that they v 'ere attempting to respond to 
the review while also developing voluminous 
materials related to the HUD grant. Further, they 
felt that the review panel and CAC were ''out to 
get them." 

The catchment area council accepted the 
report of the review panel and forwarded it to the 
regional mental health board. On the day of the 
RMHB meeting, the Keystone House staff made 
available more recent information (for 1981) on 
utilization and cost which showed that occupancy 
in the apartment program had increased, while it 
had declined in the halfway house. 

RECOMMENDATIONS 

Based on the new information, the review panel 
met again and reaffirmed its recommendation. 
The CAC, however, recommended funding of the 
apartment program on the following conditions: 
(1) that an agency policy statement be developed 
(and approved by the CACs by September 30, 
1982) that provided for admission of clients 
directly to the apartment program; (2) that 
utilization and occupancy rates be based on four 
beds; (3) that occupancy should be 95 percent; (4) 
that efforts be made to secure space for five beds 
whan leases were renegotiated; (5) and that 
measurable program goals and objectives be 
developed and submitted to the CACs. The terms 
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were met by the agency by the end of September 
1982. 

Specific recommendations included: that a log 
of telephone calls be kept by the agency in order 
to ensure that vacancies were promptly filled, and 
that Keystone clients increase their utilization of 
other local mental health resources. 

STEPS TO ENSURE IMPLEMENTATION 

Keystone was required to submit a report to the 
CACs on its policies concerning direct admission 
to the halfway house, specific measurable goals 
and objectives, and documentation of the 
95 -percent occupancy rate. These conditions are 
also incorporated into the letter of award from 
the department of mental health. Information 
about the review was published periodically in the 
local press, and meetings at which the review was 
discussed were public. 

EXTENT OF IMPLEMENTATION 

Most of the conditions have been met. The 
agency will be expected to report periodically to 
the CACs and RMHB regarding the occupancy 
rate in the apartment program and efforts to find 
new apartments. Quarterly utilization reports are 
also submitted by all grantees to the RMHB and 
regional director which show sources of referral 
and number of clients in residence. 

SPECIAL BARRIERS OR SUPPORTS 

There was very active involvement from all 
who participated ':n the review. However, there 
were personality and conceptual clashes between 
the members of the review panel and the CACs on 
the one hand, and the agency board and staff on 
the other. In part this may have resulted from the 
extensive ajiiount of time that implementation of 
the HUD grant required and the perception by 
agency representatives that the reviewers were 
operating unconstructively. The review panel had 
difficulty obtaining materials in a timely fashion 
and were concerned that the apartment program 
was not a cost-effective expenditure of scarce 
grant funds. In addition, there was a major change 
in the agency's board membership in the course of 
the review, which temporarily hampered the 
ability of these new, somewhat inexperienced 
members, to participate fully and clearly in the 
review process. The lack of o.^uable information 
on the most recent year's costs was problematic 
for the review committee, and its submission n 
the day of the RMHB meeting left little alter- 
native but to refer the report back to the CACs. 

The lack of reliable comparable data on 
staffing, costs, grants, and utilization of other 
halfway houses throughout the State and the lack 
of a policy within the department of mental 
health regarding funding of halfway houses and 
apartment programs were also constraints. 



The existence of a defined agency review 
procedure and a format for review reports was 
helpful in structuring the review process. Other 
facilitating factors included the recognition 
among volunteers on the catchment area councils 
and regional mental health board of the critical 
need for housing programs for discharged 
psychiatric patients, the desire of the volunteers 
to avoid defunding an existing program, their 
commitment to doing the right thing, and their 
acceptance of the final decision of the regional 
board. 

RESOURCES AND COSTS 

It is difficult to estimate exactly the amount of 
volunteer and staff time involved. There were 
four meetings of the panel, two CAC meetings 
involving 16 people, two RMHB meetings 
involving up to 16 people, and innumerable hours 
of staff time in developing the information, 
drafting reports, responding to agency requests, 
evaluating information, etc. This review took 
about a year from start to finish, which is almost 
double the time a review ordinarily takes. In 
terms of RMH3 staff time, a cost of about $3,000 
is an approximate figure. If the value of time 
offered on a volunteer basis is estimated, and $30 
per hour is considP' .d the average value otr time 
offered, then a minimum of $3,000 of time was 
used. This does not include the costs of repro- 
ducing reports, or the time used by agency staff 
and board, or the regional director's office. 

ADDITIONAL COMMENTS 

It probably would make sense to conduct an 
orientation with grantee agencies on some regular 
basis, perhaps annually, on the goals and methods 
of the review process. Agencies may respond 
defensively because they are frightened of losing 
funding. Reviewer* are caught between their dual 
role of advocating and sustaining services for the 
mentally ill, and fulfilling their review mandate 
by undertaking substantial and thorough evalua- 
tions. There are bound to be some occasions when 
these conflicting requirements pit agencies and 
reviewers against one another. The emotional 
reactions which can result may cloud the 
fundamental issues of provision of service and 
constructive criticism. 

The Keystone board of directors felt con- 
strained by their legal advice concerning the 
acceptance of outside cliants into the apartment 
program. The CACs and regional board felt that 
they were paying for five beds when only four 
were being used. Ultimately, both sides com- 
promised; the RMHB agreed to fund four beds at 
least for the immediate future, and Keystone 
agreed to accept outside clients into the apart- 
ment program. Though there was discomfort on 
all sides, an acceptable accommodation was 
negotiated among all parties. 
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Finally, it is important that volunteers also be 
oriented and trained :;oncerning what is expected 
of them during the review process. Review and 
evaluation require knowledge of program content 



and cost, as well as an understanding of the 
politics of interagency relations. Finally, these 
tasks require courage to raise and deal with 
thorny issues. 



4. Volunteers Assess the Operations of a Halfway House 
Program Serving the Chronically Mentally ill 



Jarrett W. Richardson j 

Thomas Group, Inc., Minneapolis 



SUMMAR Y 

The Thomas Group board of directors charged a 
subcommittee with the task of evaluating the 
^^^cee to which the Thomas House Halfway House 
program was meeting its stated goals. The 
implicit task of the committee was to identify 
areas ot strength and weakness and bring to the 
board's attention recommendations for further 
program development. 

TYPE OF ORGANIZATION 

Thomas Group, Inc. is a private, nonprofit 
organization governed by a volunteer board of 
directors made up of consumers and relatives of 
consumers, professionals working in the mental 
health field, and civic leaders with an interest in 
mental health. The primary purpose of the board 
is to create and operate adequate transitional 
living services for the area's recovering mentally 
ill. There are 21 board members nominated from 
the cr:nmunity and elected to membership by the 
board of directors. Normal responsibilities for the 
board of directors include oarticipation in 
monthly board meetings as well as participation in 
a variety of subcommittees such as personnel, 
admissions, finances, executive, and evaluation 
committees. 

EVALUATORS OR MONITORS 

Three members of the Thomas Group board of 
directors— a housewife, a minister, and a psychi- 
atrist—were appointed to the program evaluation 
committee. 

REASONS FOR EVALUATION 
OR MONITORING 

This effort was prompted by the desire of the 
board of directors to evaluate the degree to which 
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Thomas House was meeting its stated goals after 
more than Z full years of operation. This was to 
be an in-house evaluation for the purpose of 
assessing and improvir i the Thomas House 
program. This is in addition to an ongoing 
evaluation for each individual resident that was 
instituted at the beginning of Thomas House 
operation by the board of directors and carried 
out by the Thomas House staff. 

LEVEL OF PARTICIPATION 

This entire evaluation was carried out by the 
three members of the program evaluation 
committee with the assistance of the director of 
Thomas House Halfway House, who has some 
expertise in conducting evaluations. The evalua- 
tion also benefitted from a single consultation 
session volunteered by an independent profes- 
sional planning consultant. The committee 
conceived, designed, and organized an evaluation 
instrument. They also tabulated and interpreted 
the data and prepared a comprehensive report for 
the board of directors. Clerical assistance was 
provided by the staff of Thomas House but 
tabulation, interpretation and report writing were 
done entirely by the committee. 

TARGET OF EVALUATION OR MONITORING 

This evaluation focused on Thomas House, a 
3-year-old, 16-bed transitional living house that 
serves individuals who spend approximately 6 
months to 1 year learning or relearning indepen- 
dent living, socialization, leisure, and community 
survival skills. The survey included all names 
submitted by members of the Thomas House 
board, all present and past Thomas House 
residents, all Thomas House board members, ail 
present and past Thomas House staff, and 
families, and personnel from community mental 
health, social welfare, employment and educa- 
tional organizations who had significant contact 
with Thomas House and its residents. Thomas 
House Halfway House serves Olmstead County, 
which has a population of approximately 80,000. 

The survey was conducted during that period of 
time when Rochester State Hospital, which was 
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the referral hospital for the State hospital 
system, was being closed as a result of State 
budget action. The presence of the Rochester 
State Hrjpital had led to a very stable and 
progressive public sector within the mental health 
community, and that hospital had been the 
primary source of referrals for Thomas House. In 
spite of the fact that the mental health com- 
munity was in a significant state of flux during 
the time of this evaluation, the extensive 
community support for Thomas House and high 
level of interest in the activities of Thomas House 
among health care professionals resulted in a good 
response to the evaluation effort. 



PROBLEMS OR ISSUES EVALUATED 
O R MONITORED 

This evaluation focused on the stated program 
purposes of Thomas House Halfway House. More 
specifically, the programing needs addressed were 
focused on attitudes, knowledge, and skills of 
present and former residents with respect to 
financial, leisure, social, homemaking, personal 
hygiene, illness management, and educational or 
employment ac^.ivities. The evaluation was an 
effort to assess ^he effectiveness of the program 
in achieving its stated goals as reflected in 
knowledge, attitude, skills, and experience of the 
present and former residents. 

TECHNIQUES USED 

The committee reviewed all available public 
documents that contained statements concerning 
the Thomas House program. This included the 
articles of incorporation, the bylaws, requests for 
funding and program description under Minnesota 
Rule 22, brochures, and public documents such as 
newspaper articles. There were more than 65 
different documents reviewed for mention of the 
Thomas House program. A list of approximately 
73 assertions were obtained and classified into 
major areas of program focus. These statements 
were condensed into 23 client-centered goals and 
a survey format was designed. This survey was 
de-^gn^j as a general document, applicable to a 
broad range of people with a great deal of 
variation in their level of involvement in the 
Tnomas House program. The survey requested 
that the various respondents assess the agency's 
effectiveness in meeting its avowed goals of 
improving clients* pcif ormance in 25 different 
aspects of community living. 

A single mailing of the Thomas House survey 
with an attached cover letter and self-addressed 
stamped envelope was done in the summer of 
1981. The program evaluation committee 
subsequently divided respondents into groups with 
common characteristics and analyzed the 
responses of each group. Respondent groups were: 
current residents, former residents, parents of 
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residents, social workers, Rochester State 
Hospital employees, psychiatrists, psychologists, 
educational instructors, Thonr^^s House employees, 
and board members. 

FINDINGS OF EVALUATION 
OR MONITORING 

Forty-eight percent of the sample responded to 
the questionnaire. The committee interpreted the 
overall results of the survey as demonstrating 
Thomas House's effectiveness in improving client 
performance. However, present and former 
residents perceived the fulfillment of almost all 
program goals in a slightly less favorable light 
than did all other groups of survey respondents. 
Specific areas of greatest weakness as perceived 
by present and former residents were: (1) buying 
and taking care of clothes; (2) managing personal 
finances and using community financial resources; 
and, (3) using community spiritual resources. The 
committee noted the difficulty of understanding 
what the term "community spiritual resources" 
meant to present and former residents (there is no 
statement in the Thomas House program docu- 
ments concerning spiritual aspects of the 
program). However, since concern was expressed, 
this area was thought to deserve further clari- 
fication in the future. 

RECOMMENDATIONS 

The program evaluation committee of the 
Thomas House Board commended the board of 
directors of Thomas House for achieving the vast 
majority of its stated program goals and asked 
that this be communicated to the staff. The 
committee recommended that specific attention 
be given to the areas of clothing and financial 
management in future program planning and also 
recommended that further clarification be given 
to ways in which spiritual/religious needs of 
residents are met within the program or to what 
degree the program staff could appropriately 
address those needs. The committee recom- 
mended that at some time in the future a more 
specific program evaluation directed primarily at 
former residents and their families should be 
conducted. 



STEPS TO ENSURE IMPLEMENTATION 

Since this was an in-house evaluation made to 
the board of directors and the staff of Thomas 
House Halfway House, Implementation measures 
were the direct responsibility of the board. 

EXTENT OF IMPLEMENTATION 

Immediate channes were forthcoming in the 
Thomas House program as recommended by the 
evaluation committee. 
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SPECIAL BARRKRS OR SUPPORTS 

The primary difficulty with this evaluation was 
designing an instrument that could be used by a 
wide variety of residents and other people 
associated with Thomas House. An additional 
problem was that of assuring an adequately high 
response rate for the survey. Because of the broad 
spectrum of activities involved in the Thomas 
House program, detailed assessment of every 
aspect of the program would have necessitated 
too cumbersome a document for careful response 
from a wide range of people. The resulting 
assessment of broad categories of program 
functioning may have overlooked some significant 
areas, although the opportunity for specific 
comment was provided and a number of respon- 
dents took advantage of that. Data collection 
efforts were enhanced by the widespread 
community pride in the existence of the Thomas 
House program and the broad base of financial 
support. The most difficult barrier was the closing 
of the Rochester State Hospital and its disturbing 
resultSw This introduced a significant number of 
variables into what had been a stable service 
delivery system. 

RESOURCES AND COSTS 

The major expenses for this study were 
photocopying and poctage expenses, and therefore 
the monetary cost was relatively small. Total 
staff and committer hours involved in the 
evaluation are estimated at i50 person-hours. 



Because of the volunteer nature of the program 
evaluation committee and the significant 
involvement of those committee members in 
other community service and professional 
activities, the time required for initiating, 
executing, and interpreting this evaluation was 
prolonged. However, the committee and the 
Thomas House board of directors are gratified 
with the results of this effort. 

ADDITIONAL COMMENTS 

This evaluation process was valuable to the 
committee members and the board of directors 
from an educational standpoint. It improved the 
level of knowledge that participants had about the 
program objectives and the operation of the 
program. It does seem that the effort involved in 
organizing, designing, producing, and carrying out 
such an evaluation consumed an inappropriate 
amount of volunteer time. If funding were 
available for such an evaluation to be done or 
coordinated by a paid professional, the board of 
directors and evaluation CDmmittee cot id obtain 
similar benefits with fewer volunteer hours. In a 
set^.mg in which volunteer efforts and services are 
at a premium and need to be widely used, it would 
be our impression that having a professional 
consultant actually carry out the evaluation 
process with the assistance and participation of 
board members would be an optimal arrangement. 
Having volunteers performing technical tasks for 
which they were not trained was the weakest part 
of the process. 



5. External Group Advocates for Better Cai & 

Darlene Humphrey 

Consumer Advocates for Better Care, Leominster, Massachusetts 



SUMMARY 

Consumer Advocates for Better Care (CABC) 
provides daily visitation to all nursing and rest 
homes in the north Worcester County, Massachu- 
setts area. CABC advises residents of their rights, 
and receives, documents, investigates, and 
attempts to resolve all complaints received from 
residents and/or their families. In a recent c?.se, 
CABC was successful in bringing to light abuses in 
a rest home. The ultimate result was the closing 
of the facility. 

TYPE OF ORGANIZATION 

CABC originated in 1975 when a group of 
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elderly citizens contacted the Herbert Lipton 
Mental Health Center to discuss their concerns 
about nursing homes in the Fitchburg, Massachu- 
setts area. They were interested in supporting 
legislation to improve the care provided in nursing 
homes, and in visiting and evaluating nursing 
homes. In addition they hoped to stimulate public 
awareness about nursing homes and to advocate 
for the rights of residents. In 1977, with the 
assistance of tlie mental health agency, they were 
awarded a grant from the Administration on 
Aging to become one of the six model advocacy 
projects nationwide. The money supported a 
part-time director who was responsible for 
program implementation, volunteer recruitment 
and training, preparation of forms to document 
investigations, and complaint resolution. 

Initially two advocates were hired through the 
Green Thumb program (funded by the Federal 
Community Service Employment Act, Title V of 
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the Older Americans Act) to visit area homes and 
to monitor the care received by residents of these 
homes. Subsequently, the program received 
funding through the area mental health agency to 
hire the director full time. Offices and telephones 
were donated by the mental health agency. In 
1979, CABC became a designated nursing home 
ombudsman program in Massachusetts and was 
able to submit a proposal to the Region II Area 
Agency on Aging for Federal funding under Title 
III of the Older Americans Act. This enabled 
CABC to fund a full-time office, telephones, 
travel, supplies, etc. CABC is governed by a board 
of 12 community members including consumers, 
lawyers, and personnel from other agencies 
concerned with problsms of aging. 

EVALUATORS OR MONITORS 

Presently CABC has 12 visiting ombudsmen 
ranging in age from 22 to 77 years. The majority 
of the advocates are low-income older persons 
who are recruited by the director and paid by 
either Federal Title V funds (Senior Aids and 
Green Thumb) or another Older Americans Act 
program, the Elder Service Corps, which provides 
small stipends for such purposes. The majority of 
our advocates are lay persons who have had 
extensive training in issue advocacy, the aging 
process, characteristics of nursing homes, law and 
aging. Medicare and Medicaid, and who receive 
ongoing monthly training as needed. All advocates 
are now certified by the state as ombudsmen. 
When the CABC ombudsmen began visiting rest 
homes, they could see there was a need to become 
better informed as to the special needs of elderly 
persons who had been deinstitutionalized from 
state hospitals. We contracted with our local 
mental health agency for training in the uses of 
psychotropic medications and their side effects, 
improving communications better with deinsti- 
tutionalized residents, and responding to their 
special needs. 

REASONS FOR EVALUATION OR MONITORING 

Residents of nursing and rest homes need 
interested community members to advocate for 
their rights since 75 percent of residents of 
nursing and rest homes have no family or friends 
to advocate for them. Regulations must be 
enforced and new legislation enacted to improve 
the lives of institutionalized older .persons. 
Quality of care has improved somewhat because 
of community involvement but there is much 
more that the community could do to improve the 
lives of old people in nursing homes. Many people 
forget that residents of nursing homes are stiil 
part of the community. 

^EVEL OF PARTICIPATION 

Volunteer advocates in the CABC program 



function as independent and external monitors of 
nursing home programs. 

TARGET OF EVALUATION OR MONITORING 

Yhe agency's 12 visiting advocates monitor the 
care received by residents of nursing homes and 
rest homes in 22 communities. There are 41 
nursing and rest homes in north Worcester County 
with a total population of 1,874 residents. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

CABC advocate visits have uncovered many 
abuses of residents' rights ranging from com- 
plaints of cold or improper food to serious abuse 
and neglect. In one recent case involving a rest 
home, CABC advocates documented a range of 
abuses including lack of nutritious meals, phyi»ical 
abuse, withholding of medication for punishment, 
denial of leisure activities, and misuse of resident 
allowances. 

TECHNIQUES USED 

The technique used to monitor is daily visita- 
tion to nursing and rest homes. The visits are also 
used to gain the confidence of residents and to 
persuade them that advocates can resolve their 
complaints and that ople ir. the community 
care. Advocates become friends of nursing home 
residents and residents become confident that 
they can share their concerns without fear of 
retaliation from nursing home administration or 
staff. Brochures are handed out to each resident 
and their fanralies. Community forums and 
speaking engagements are conducted to educate 
the community as to residents* rights. CABC 
volunteers and staff appeared on television and 
radio talk shows. In 1976, CABC joined the 
National Citizens Coalition for Nursing Home 
Reform in order to have a greater impact upon 
national legislation. CABC*s director served two 
terms as vice president of the national coalition. 

FINDINGS OF EVALUATION OR MONITORING 

Based on information gathered by CABC, the 
rest home cited above became the subject of 
litigation initiated by the Massachusetts attorney 
general. In its request for a preliminary injunction 
against the home, the Commonwealth offered the 
following findings regarding deficiencies: 

• Failure to provide at least three dally meals 
that are nutritious, that meet the minimum 
dally food allowances set by the Food and 
Nutrition Board of the National Research 
Council, adjusted for age, sex, and activity, 
and that are suited to the special needs of 
residents. Including residents on restricted or 
therapeutic diets 
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• Failure to administer medications to 
residents in dosages and at times prescribed 
by current physicians' ordars 

• Failure to keep receipts for all expenditures 
of one dollar or nr^ore made on behalf of 
residents out of their personal needs accounts 

• Admittance of prospective residents who 
require medications which can be adminis- 
tered only by trained nursing personnel whom 
the rest home has not employed 

• Failure to clean bathrooms, living areas, 
bedrooms, kitchen areas, dining areas, 
storage areas, attics, and cellars; and to 
provide all necessary cleaning supplies and 
materials to maintain these areas in a safe 
and sanitary manner 

• Failure to allocate and budget funds to be 
used for resident activities, hobbies, trips, 
crafts, reading materials, and games 

RECOMMENDATIONS 

The attorney general's initial recommendation 
to the court regarding this specific case was to 
enjoin staff of the rest home from continuing any 
of the abuses outlined in the complaint, 

STEPS TO ENSURE IMPLEMENTATION 

CABC has testified for legislation at the State 
and national level. They have also informed the 
public about needed legislation and asked for their 
support. CABC also works very closely with the 
local lecial services agency, and public health and 
mental health agencies. In the case of the rest 
home, CABC turned its information over to legal 
services whose staff in turn obtained depositions 
from aides at the home. 

EXTENT OF IMPLEMENTATION 

The rest home doors have now been closed and 
its license has been suspended. Mental health • 



clients have been placed in other homes by CABC 
advocates themselves. 

SPECIAL BARRIERS OR SUPPORTS 

CABC's biggest barrier to the implementation 
of the advocacy program was, and still is, access 
to nursing homes. There is a bill pending at this 
time that would give community groups access to 
nursing and rest homes. Another barrier is that 
volunteers are very difficult to recruit. Nursing 
homes do not have the pleasant atmosphere one 
wishes they had. People are also afraid of their 
own mortality. They find visits to nursing homes 
very depressing. More community education is 
needed to help change these negative attitudes 
about elderly, infirm persons. CABC's biggest 
support comes from the dedicated people who 
work with our program. They care about the cause 
and have worked diligently to try to improve the 
lives of nursing home residents. The CABC board 
should be commended for the support they have 
given through the years. 

RESOURCES AND COSTS 

CABC employs a full-time director and 
halftime assistant and halftime secretary for a 
total staff cost of $26,000. Volunteer or Title V 
time amounts to 240 hours per week. CABC is 
jiow funded by the Region II Area Agency on 
Aging with Federal Title III funds. We rei:eive 
$1,500 from our United Way and rely on founda- 
tion money to help defray many costs. 

ADDITIONAL COMMENTS 

Consumer groups such as ours must have more 
interest from staff members and family and 
friends of nursing home residents. Complaints are 
received in our office anonymously and we cannot 
act upon them. People do not want to become 
involved. Nursing home care will never improve 
substantially without involvement of the commu- 
nity. Hopefully, the day will come when there will 
be no cases submitted as serious as the one I have 
included in this report. 



6. A Mental Health Advocacy Program Evaluates 
an Agency's Programs and Procedures 

Nancy Sohlberg 

Site Visitation Committee of the Mental Health Association of Illinois Valley 



SUMMARY 



The site visitation committee of the Mental 
Health Asso ciation of Illinois Valley (MHAIV) 

For further information write Nancy Sohlberg, 3945 
49th Avenue South. St. Petersburg. FL 33711. (813) 
864-1809. 



25 



ERLC 



evaluated the programs and service delivery of 
the Human Service Center of the Peoria Area, 
Inc. (HSC), The programs monitored included: 
sustaining care, outpatient mental health, drug 
and alcohol, outreach, emergency response, and 
consultation and education services. Two 
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residential facilities for substance abuse, a 
halfway house for the mentally ill, and a family 
home for teenage girls were included. Client 
satisfaction and community awareness were 
assessed. The site visitation committee presented 
a final report at a meeting with the Human 
Service Center executive director and the 
program committee and reviewed their findings 
and recommendations with the president and 
members of the board of directors of the Human 
Service Center. 

TYPE OF ORGANIZATION 

Members of the site visitation committee were 
recruited by the Mental Health Association of 
Illinois Valley advocacy chairman because of their 
individual skills or interest in evaluating specific 
mental health programs. MHA;v is a voluntary 
organization serving three counties. It provides 
programs in crisis intervention, education, and 
advocacy. The advocacy program, in addition to 
conducting site visits, has direct responsibility for 
advocacy to clients of the Human Service Center, 
The George Zeller Mental Health Center, and five 
area nursing homes. The organization of MHAIV 
consists of an executive director, a 28*member 
board of directors elected for a 3-year term and a 
professional advisory board. 

EVALUATORS OR MONITORS 

The site visitation committee was composed of 
a chairperson and 12 members. It included a 
psychiatrist, a professor of pharmacology, two 
psychologists, a counselor, three nurses, the crisis 
intervention chairman for MHAIV, a nursing home 
administrator, a health educator, a teacher, and a 
lawyer. None of the members were employed by 
the HSC. Ten were MHAIV board members. 

REASONS FOR EVALUATION OR MONITORING 

In accordance with its purpose as a voluntary, 
citizen-based, and consumer-oriented organiza- 
tion, the Mental Health Association of Illinois 
Valley and the Human Service Center of the 
Peoria Area signed a statement of cooperation in 
1979. This authorized MHAIV to provide advocacy 
services for HSC clients and to conduct biannual 
reviews of policies, procedures, confidentiality, 
and utilization of data concerning the HSC*s 
programs and services. The regional health 
systems agency had also expressed interest in 
such a review. There have been two evaluations of 
the HSC, in September 1979 and 1981. 

LEVEL OF PARTICIPATION 

Committee members were actively engaged in 
every phase of the evaluation. The committee 
requested and reviewed pertinent data, e.g., 
reports of the most recent site visits by the 
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National Institute of Mental Health (NIMH), 
Department of Health and Human Service*?, and 
Illinois Department of Dangerous Drugs. Budgets, 
fee scales, procedure books, and contractual 
agreements with community agencies were 
reviewed. The committee attended an informa- 
tional slide presentation given by the HSC 
executive director and program director and 
worked closely with staff and bc^rd members, 
arranging interviews with staff and scheduling 
site visits at appropriate times. They reviewed 
their findings and recomrnenaations and wrote a 
report on each program. The reports were 
presented to the executive director and board of 
directors of the HSC. 

TARGET OF EVALUATION OR MONITORING 

The Human Service Center of the Peoria Area 
is a comprehensive community mental health 
center incorporatad in 1976 through consolidation 
of a number of different agencies. The center 
offers a wide range of services in crisis inter- 
vention, outpatient and sustaining care, substance 
abuse, and education and prevention. Facilities 
are located throughout the Peoria area with 
satellite offices in neighboring communities. 
Other mental health services are provided in 
inpatient psychiatric units in two general 
hospitals, clinics, at the Peoria School of 
Medicine, and in a Veterans Administration clinic. 
A detoxification unit and a chemical dependency 
center are available. The HSC has an executive 
director, a board of directors, and a citizens 
advisory board. Peoria is a medium sized indus- 
trial city with rising unemployment. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The committee's task was to evaluate each 
program with respect to: intake, discharge^ 
referral, and followup procedures; staffing 
patterns; modalities and goals of treatment; 
physical facilities and accessibility; patients' 
rights and client satisfaction; fee policies; staff 
development and attitudes; and community 
awareness and attitudes. An earlier NIMH 
evaluation had noted that the agency was not 
visible in the community, that the board was not 
familiar with the requirements for a community 
mental health center, and that there needed to be 
a full-time qualified director for the education 
and prevention program. In the 1979 visit, MHAIV 
had recommended several alterations, refur- 
bishings, and increased maintenance at the 
alcohol residential center. These issues suggested 
areas for further evaluation. 

TECHNIQUES USED 

To estimate client satisfaction, a short 
questionnaire was distributed to the clients 
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present at each program site visit. Another 
questionnaire was mailed to two organizations: 
the Alliance foi the Mentally III (a support group 
for the families and friends of mental patients) 
and Recovery (a self-help group for emotionally 
disturbed persons). To determine community 
opinion, individual letters were sent to the 
agencies contracting for HSC program services. 
Interviews \KZTe conducted with key personnel at 
Zelle^ Mental Health Center to assess the 
effectiveness of the sustaining rare program and 
the appropriateness of referrals to Zeller by the 
emergency response service. The chief of police 
was contacted concerning the utilization of the 
emergency response service. The recommenda- 
tions made by previous site visits determined our 
main focus. Each committee member formulated 
his/her own questions and designed a site visit 
strategy. In general, these strategies were based 
on the following sources: Citizen Evaluation of 
Mental Health Services, (McMurray et al. 1976), A 
Trust of Evaluation {Zinober and Dinkel 1981), and 
CMHC Requirements and Definitions of States 
(1980). To test board member • awareness, the 
chairman addressed a board meeting explaining 
the purpose of the evaluations and gave members 
a short v^itten quiz to assess their familiarity 
with the CMHC requirements. The site visitation 
committee worked closely with the HSC program 
committee C'*^ invited them to go along on a 
visit. The site visits themselves were scheduled 
over a 2-week period at the following programs: 

• Sustaining care 

• Outpatient care 

• Alcohol and drugs 

• Emergency response service and status 
offenders 

• Education and prevention program 

FINDINGS OF EVALUATION 
OR MONITORING 

The site visitation committee was generally 
very satisfied with the Human Service Center's 
services and noted considerable growth in content 
and scope. The high degree of professionalism, 
better utilization of staff, and the use of 
part-time staff for evening hours allowed 
increased evening programming without hiring a 
large number of additional staff. Other improve- 
ments included: 

• Inauguration of a senior world daycare 
program that the committee felt was 
meeting community needs 

• Establishment of a general policy for fee 
collection 
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• Improvements in the alcohol program 
residential services including: structural 
changes, improved parking, and improved 
interior decoration 

• Hiring a well-qualified director of education 
and prevention and instituting a broad range 
of programs for professionals, clients, and 
the community 

Client responses to the questionnaire suggested 
good accessibility and satisfaction with services. 
Community agencies responding to their ques- 
tionnaires indicated a high level of trust and 
cooperative working arrangements. 

On the negative side, the site visitation 
committee found that there was still a need for 
halfway house beds for alcoholic women; kitchen 
equipment and new furniture were needed in the 
drug residential center; and carpeting needed to 
be replaced in the alcohol treatment center. 
There was an apparent discrepancy in fees for 
education programs offered to the public that 
might make them unaffordable to some segments 
of the community. It was also noted that the 
community at large was not very responsive to 
providing housing for the mentally ill or geriatric 
daycare programs. Finally, because of a declining 
Federal grant, funding continued to be critical 
and it was vitally necessary to obtain community 
support for the emergency response service. 



RECOMMENDATIONS 

The Human Service Center was commended for 
its considerable progress since 1979 in the quality 
and scope of programs offered. The improved fee 
collection statistics, excellent utilization of staff, 
and efforts of the director, staff, and board to 
heighten awareness and visibility of the agency 
within the community were striking examples of 
improvement. The site visitation committee made 
some recommendations and suggestions in its final 
report. These included: 

• Drug residential center should replace broken 
furniture, add kitchen equipment, and try to 
obtain some dental care for clients (The staff 
had also requested additional psychiatric or 
psychological staffing) 

• Additional halfway house beds for female 
alcoholics and replacement of worn furniture 
and carpeting in the residental unit 

• The director of sustaining care cited a need 
for better coordination with referrinc sources 
(e.g., mental health center and hospitals) 

• Since sustaining care clients fite usually in a 
low-income bracket, staff suggested a 
possible reduction in their fees 
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♦ Fees for education and prevention programs 
should be reassessed to make them more 
affordable and competitive 

♦ Continued striving for better visability and 
credibility in the community is essential 

♦ The declining Federal grant money makes it 
vital to enlist the community's financial 
support for the emergency response service 

STEPS TO ENSURE IMPLEMENTATION 

Evaluating service delivery every other year 
gives agencies time to respond to recommenda- 
tions. Since the committee had access to site visit 
recommendations from the National Institute of 
Mental Health and Department of Health and 
Human Services it was able to follow up on those 
suggestions as welL The committee worked 
directly with the HSC program committee, 
making suggestions directly to them. A 
relationship of trust was established, which 
facilitated communication and increased the 
likelihood of change. The HSC director and board 
were very attentive and appreciative of the 
committee's findings and recommendations. 

EXTENT OF IMPLEMENTATION 

Several recommendations of the 1979 site 
visitation were implemented. Structural 
improvements in the alcohol unit increased its 
safety, attractiveness, and visibility. In the drug 
facility, safety was upgraded through physical 
alterations and security for medications was 
improved. A patients' rights statement for HSC 
was translated into Spanish and Vietnamese by 
MHAIV board members to meet the needs of the 
client population. An emergency personnel change 
was made as a result of a recommendation from 
the chief of police. The HSC executive director 
and board president took thb site visit evaluation 
to the health systems agency and presented it at 
the Unite^ Way allocation hearings. Mental 
Health Association board members have also 
attended city council hearings in support of the 
emergency response service. Preliminary 
meetings between the medical director of Zeller 
Mental Health Center and sustaining care staff 
have begun. The Human Service Center recently 
opened new HUD housing for mentally ill clients. 



SPECIAL BARRIERS OR SUPPORTS 

Certainly the interest and energy invested by 
the site visitation committee was responsible for 
such a comprehensive, consumer-oriented 
evaluation. Further, the suppor: and cooperation 
of the executive director, board of directors, and 
staff of the Human Service Center cannot oe 
overestimated. The con'trrattee was not entirely 
satisfied with the method of reviewing records, 
although one member did attend record review 
meetings and a small number of records were 
reviewed. The main benefits derived from the site 
vi:its has been the increased awareness, respect, 
and the climate of cooperation that has developed 
between the MHAIV and the HSC. 



RESOURCES AND COSTS 

One month was required to complete the 
evaluation from the planning through the writing 
and presentation of reports. Some committee 
members participated in only one site visit while 
others contributed to two or three program 
evaluations. The best estimate is that volunteer 
time totaled 160 hours. Four days of clerical time 
were required to type and mail letters, ques- 
tionnaires, and final reports. The cost of paper, 
mailing, and duplication amounted to $80 and was 
covered by he advocacy budget. HSC duplicated 
and mailed final reports to all HSC board 
members. Volunteer time required in the second 
evaluation was considerably less because of basic 
knowledge and skills developed during the 1979 
evaluation. 

ADDITIONAL COMMENTS 

The Mental Health Association of Illinois Valley 
is very pleased to have completed two evaluations 
of the Human Service Center. Committee 
members enjoyed the learning experience, if not 
the actual report writing. Since the advent of the 
block grant system and reduction of its original 
grant, the HSC has had to make many adjustments 
in staffing an^ programing that will probably 
demand many changes in the next program 
evaluation. The committee urges other advocacy 
groups or other mental health associations to 
involve themselves in such an interesting and 
rewarding experience. 
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7. Mental Health Center Board Increases Involvement in 
Evaluation of Center Needs 



Roger Strauss 



Washington County Mental Health Services, Inc, 
Montpelier, Vermont 



SUMMARY 

The standards and evaluation committee of 
Washington County Mental Health Services, Inc., 
(WCMHS) conducted an assessment of the mental 
health needs of the community and the capability 
of the mental health center to meet those needs. 
This project was important because it occurred 
during a time of budget cuts and diminishing 
resources. The committee members solicited 
information from community representatives and 
center staff regarding needed services and the 
priority for such services in case of: (I) an 
increase in funding. (2) a decrease in funding, (3) 
same level funding. The results were compiled and 
submitted to the full Board of WCMHS which then 
ranked the services according to priority. When 
funding projections vere clarified the following 
year the board and the executive director were 
able to make decisions and plan for the future 
based on the findings from the project. 

TYPE OF ORGANIZATION 

Washington County Mental Health Services is a 
private, nonprofit, community mental health 
cent ^r with a governing board of up to 25 citizens 
with an interest in mental health. The function of 
the board is to set policy for WCMHS, to be 
responsible for management, and to hire and fire 
the executive director. Part of the board's 
general responsibility is to monitor programs 
against criteria set by others, including the 
Vermont Department of Mental Health, the 
National Institute of Mental Health (NIMH), and 
the National Institute of Alcohol and Alcohol 
Abuse (NIAAA). To do this, the board formed the 
standards and evaluation committee. 

EVALUATORS OR MONITORS 

The standards and evaluation committee was 
made up of both board and staff of the center, 
and was chaired by the executive director of the 
center . One of the committee members was a 
minister, another was the executive director of a 
local United Way, two were housewives, and one 
member was a school bus/ambulance driver. The 
only professi onal on the committee was a college 

For further information write Roger Strauss, Executive 
Director, Washington County Mental Health Services. 
Inc., P.O. Box 647, Montpelier, VT 05602, (802) 
229-0591. 
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teacher in child psychology. There were repre- 
sentatives from both line staff and administra- 
tion. When the committee evaluated a program in 
which one of the committee members was also a 
staff nr^ember, that person did not participate. 

REASONS FOR EVALUATION OR MONITORING 

Monitoring .— Every year the Vermont Depart- 
ment '^f Mental Health came to each center in the 
state to do a 3-day site visit. Sometimes as part 
of that visit, sometimes separate, a representa- 
tive cf NIMH would also make a site visit. 
External monitors from such organizations as 
NIAAA, the local United Way organization, and 
the local district advisory health council also 
made site visits. 

In the early 1970's, the executive director 
monitored all the recommendations from the site 
visits to ensure they were implemented. The only 
input of the board was to respond to site visit 
reports. To increase board involvement, it was 
suggested that board members and staff jointly 
form the standards and evaluation committee 
which would be responsible for monitoring and 
evaluating programs in light of the recommenda- 
tions from external monitors. 

Planning .— Durino the 1980 Vermont legislative 
session the Community Services Act was passed. 
This bill mandated that mental health centers 
have a plan of service delivery available for 
public review. This necessitated a community 
needs assessment by each center. Because the 
members of the standards and evaluation 
committee had been intimately involved in 
monitoring programs at WCMHS, the board felt 
that they would be the natural team for coor- 
dinating and organizing the needs assessment and 
subsequent plan for the center. 

LEVEL OF PARTICIPATION 

Board members of the standards and evaluation 
committee were involved with center staff in 
every stage of program monitoring and planning 
except for interviews with community profes- 
sionals. Volunteers were excluded from this task 
because of an assumption that community 
professionals would not be candid with WCMHS 
staff if a board member were present. In retro- 
spect, this decision was an error. Community 
professionals seemed eager to participate in the 
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needs stuo^ and consequently a board member's 
presence probably would not have detracted from 
the spontaneous responses. 

A board member on the standards and evalution 
team went to every program planning session to 
listen to staff express their ideas as to what they 
would do if funding increased, decreased, or 
stayed the same. 

TARGET OF EVALUATION 
OR MONITORING 

The assessment centered on the short- and 
long-term goals of WCMHS and on the capability 
of its program components to fulfill the needs as 
defined by the community. WCMHS serves a 
catchment area of about 50,000 residents. The 
area includes two small urban centers— Mont- 
pelier, which is the state capital (8,000); and 
Barre, a bluecollar community (1 1,000). A large 
number of clients (400) have been discharged from 
the State's only hospital for the mentally ill, 
which is also located in the county. Aftercare 
services to this population are, therefore, a high 
priority at WCMHS. 

WCMHS also provides a day hospital, a children 
and youth service, a substance abuse program, a 
24-hour emergency service, programs for 
mentally retarded persons, an outpatient service, 
and residential programs for mentally ill, 
mentally retarded, and autistic persons. All these 
programs were assessed by the standards and 
evaluation committee. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The evaluation and monitoring functions of the 
standards and evaluation committee only went 
into effect after a site review from an outside 
agency* Therefore the issues changed according to 
the findings of the external reviewer. Once the 
standards and evaluation committee took over the 
planning function, they concentrated on the needs 
identified by community representatives and 
program staff. 

TECHNIQUES USED 

Monitoring .— After a site visit, the standards 
and evaluation committee reviewed the recom- 
mendations of the external monitor. Members 
then broke up into teams of three, and each team 
visited each center program mentioned in the site 
visit report. The team reviewed the report with 
the program director and program staff. The first 
question was whether the recommendations were 
valid. If the program staff said the recommenda- 
tions were not justified, the team would 
determine whether the criticisms were valid. If 
the team agreed with the staff, they would then 
act as advocates for the program with the original 
monitor. 
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If the team found that the critique was valid, or 
if program staff agreed with the deficiency, a 
schedule would be set for making corrections. The 
standards and evaluation team then returned to 
the program to see if corrections had been made. 
The findings of the team were reported to the 
committee as a whole and to the executive 
director. 

Planning . — To begin the process, representa- 
tives of community agencies were invited to meet 
with the center's executive director and other 
appropriate program staff. At that meeting 
community agency staff were asked to talk about 
any improvements needed at WCMHS to meet 
future mental health needs. Minutes were sent to 
agency staff to document the discussions. These 
notes were then sent to the standards and 
evaluation committee and to all center program 
directors for distribution to staff. Committee 
members then interviewed program staff to 
determine their views of needs in light of the 
community input. Staff were asked to discuss 
ways of meeting these needs and priorities under 
three conditions: 1) same level of funding, 
2) increase in funding, 3) decrease in funding. This 
information was eventually included in the 
community needs assessment. A public meeting 
was also held to secure public input. Feedback 
from the meeting was incorporated into the needs 
document that was finally presented to the entire 
board of directors. The board then ranked 
programs anci approaches according to center- 
wide prioritias. The entire planning process took 
over a year, from the Summer of 1980 to the 
Winter of 1S81. 

FINDINGS OF EVALUATION OR MONITORING 

Monitoring .— For the most part, all site reports 
were favorable to the center in all programs. 

Planning . — Board members were impressed with 
the candid responses of community professionals 
and their openness with staff. They were also 
impressed by the overall positive responses of the 
community agencies to WCMHS. The needs 
assessment coupled with the firsthand discussions 
with program staff gave the board an excellent 
overview of the demands on the center's current 
capabilities. It also allowed the board to estimate 
what more could be done under different levels of 
funding. 

RECOMMENDATIONS 

Monitoring . — All recommendations were based 
on site visit reports. In ever> instance, the board 
and program staff reached consensus concerning 
the reviewers' reports and the time limits for 
changes. 

Planning . — The board as a whole endorsed all 
the existing programs at WCMHS and felt that all 
programs were doing important jobs. If budget 
cuts were in order, then two programs were 
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suggested for elimination. The first was a 
program for developmentally disabled pre- 
schoolers which provided rehabilitation and 
afternoon programs in their homes. It was 
vulnerable because, in 1981 a law was passed that 
encouraged loc^l sc * districts to pick up 
preschool progr^ims lur developmentally disabled 
youngsters. Efforts were n*ide to get local school 
districts to pick up this program, which eventually 
proved to be successful. 

The second was a program for older persons. In 
Washington County similar programing was 
provided by three other agencies, a retired senior 
volunteer program, a home health agency, and an 
area agency on aging. Consequently, there were 
at least some other agencies providing services, 
whereas for other populations — substance abusers, 
chronically mentally ill, and mentally retarded— 
WCMHS was the major, if not only, service 
provider. 

For the most part, the board endorsed the 
recommendations of the community and the staff. 
These recommendations can be divided into four 
major requests: (1) more consultation from staff; 
(2) moro residential care for special groups such 
as chronically mentally ill, mentally retarded, 
substance abusers, and abused spouses; (3) more 
education and prevention programs; and 
(4) quicker response time in emergency situations. 

STEPS TO ENSURE IMPLEMENTATION 

Monitoring . — Once the standards and evalution 
team met with each program under review, time 
limits were set within which necessary changes 
would be made. These dates were then communi- 
cated back to the full standards and evaluation 
committee for approval. The standards and 
evaluation task force also returned later to see if 
the changes had been made. If not, a new date for 
implementation was set. It was never necessary to 
set a third deadline for review. 

Planning .— The plan eventually became a public 
document and was to be used as a basis for any 
changes at the center. If the executive director 
deviated drastically from the plan it had to come 
before the board for review. 

EXTENT OF IMPLEMENTATION 

Monitoring . — A typical criticism was one from 
an external reviewer: "the outpatient program has 
a waiting list that is too long, according to 
Department of Mental Health criteria.** After a 
visit from the standards and evaluation commit- 
tee and a realignment of schedules and 
procedures, the outpatient department was able 
to e:>tablish a policy for seeing all clients within 
at least 7 days after their first call for an 
appointment. 

Planning . — With respect to the request for 
increased consultation and education, it was felt 
that no more time could be granted these areas 
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because of demands on the staff for direct 
services. Consequently, the board and staff set up 
a corporation called The Institute for Professional 
Practice. Staff who wished to work after hours 
for extra funds could provide courses and 
consultation. Cost of the courses and program 
would be paid by participants. Staff could make 
extra money and any profit would come back to 
the center. 

Emergency teams were given backup staff who 
could sit with patients for long periods of time, 
enabling regular emergency service staff to 
attend crisis calls much more quickly. 

The clinic negotiated with the Department of 
Mental Health to endorse a creative new resi- 
dential program for mentally ill and mentally 
retarded persons. The basic approach of the 
program calls for placing recent state hospital 
patients in heavily staffed apartments. As 
patients are rehabilitated, the staffing pattern 
becomes less intense. If clients regress, the 
staffing pattern returns to its former level. This 
allows the client to stabilize in one environment 
as opposed to being moved through a continuum of 
residences, thus eliminating the anxiety attendant 
upon multiple moves. 

SPECIAL BARRIERS OR SUPPORTS 

The board's desire to work more closely with 
staff and to get to know them better was satisfied 
through the monitoring activities. For its part, 
the staff had a chance to work as partners with 
the board in the monitoring process instead of 
their usual role as passive recipients of an 
external monitoring team's recommendations. 
This reduced the threat of the evaluation and 
increase^ tne chances that some real changes 
would be effected through the evaluation. The 
Department of Mental Health and the laws of 
Vermont provided external support for the 
planning process. The major barriers were the 
time it took for staf . and board to participate. 

RESOURCES AND COSTS 

For both staff and board, the time spent in a 
typical year on the standards and evaluation 
committee was 16 hours. This included regular 
committee meetings, v\/hich lasted about 2 hours 
each, and the task force time spent reviewing the 
site visit reports with each program. Board 
member time on the planning process was much 
less since they were not involved in the many 
interviews with the community agencies 
(approximately 5 hours each). 

ADDITIONAL COMMENTS 

The staff liked the interaction with the board 
because they felt they got to know them much 
better and they also felt themselves involved in 
an important activity of the center. They realized 
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the importance of program evaluation and made 
efforts to change their own programs when it was 
time for them to be monitored. The board enjoyed 
working with the staff. Board members also got to 
see programs from a different perspective and to 
view themselves as agents of change in the 
agency, not just passive recipients of information. 
Th3 monitoring process has a definite flaw. The 



standards and evaluation committee was only 
activated after site reviews. With the loss of all 
Federal funding there have been no site reviews 
for the past 18 months. Further, the State has not 
conducted a comprehensive evaluation of any 
center for 2 years. Therefore, the standards and 
evaluation committee has not met for nearly a 
year. 



8. Board of Visitors Monitors Service Quality 

Marilyn Seide 

Board of Visitors, Manhattan Children's Psychiatric Center 



SUMMARY 

The board of visitors of the Manhattan 
Children's Psychiatric Center was formed in 
1978—6 years after the facility opened its doors. 
The board is part of a larger network of citizen 
ov?5rseers mandated by State law and attached to 
all State mental health facilities in New York. 
The board consists of family members, interested 
lay persons, and mental health professionals. It 
meets monthly, conducts investigations of client 
abuse, reviews incident reports, and makes 
numerous site visits. The major challenge to the 
board is to maintain independence while pre- 
serving the trust and cooperation of the 
management of the center. 

TYPE OF ORGANIZATION 

The New York State Legislature established 
boards of visitors for mental hygiene facilities 
over 50 years ago. These lay boards are usually 
composed of seven people from the community 
(three of whom are parents or relatives of 
patients or former patients) who are knowledge- 
able about, and interested in, the welfare of the 
patients in the facility they represent. Members 
are responsible for oversight and monitoring of 
the quality of care and treatment in each of the 
56 mental health and mental retardation facilities 
in New York State. They are appointed by the 
Governor to serve 4-year terms, and may be 
reappointed. Members are not paid, but may be 
reimbursed for expenses incurred in carrying out 
their responsibilities. Though boards may function 
differently from each other, there are jertain 
specific tasks that they are uniformly expected to 
ce^y out. The board that plays the central role in 
this case description is responsible for monitoring 
service quality at the Manhattan Children's 
Psychiatric Center (MCPC). 



For further information write Marilyn Seide, President 
of Board of Visitors. Manhattan Children*s Psychiatric 
Center, Wards Island. New York. NY 10035. (212) 
369-0500 ext. 1001. (212) 920-7445 (home phone). 
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EVALUATORS OR MONITORS 

Though the Manhattan Children's Psychiatric 
Center was officially opened in June 1972, the 
first members of a board of visitors for the 
facility were not appointed until the spring of 
1978, when the first meeting was called by the 
director of tho MCPC. Among the original 
members of the board were a prominent meriial 
health advocate who was the ^^^^ecutive director 
of a mental health association, chairman of a 
committee on the handicappea wno was the 
parent of a child who had been an inpatient at the 
facility, two social workers with extensive 
experience in the field of children's mental 
health, a lawyer from the juvenile rights division 
of the Legal Aid Society, another parent of a 
child formerly at the center who worked part- 
time as an aide in a public school, and a mental 
health hospital administrator. Thus, through 
training, interest, experience and/or expertise, 
this board was well equipped to carry ou^ :i:s 
responsibilities as advocates for the children who 
were clients of the center. 

Rt£ASONS FOR EVALUATION 
OR MONITORIN G 

The functions of the boards of visitors are 
spelled out in State law and are generally directed 
toward assuring of the well-being of residents of 
State facilities. 

LEVEL OF PARTICIf tiqn 

Boards of visitors are both advocates a' ! 
independent watchdogs. Though they shar.^ a 
commitment to the provision of optimal patient 
care with those who are vested with the respon- 
sibility for delivering care and treatment (the 
State Offices of Mental Health, and Mental 
Retardation and Developmental Disabilities), 
board members do not always agree with those in 
the system about how care and treatment are 
being provided. This explains why the board is 
occasionally perceived by State officials as an 
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adversary. Given the board's advocacy role, such 
a perception may sometimes be unavoidable. 
However, problems can usually be worked out by 
keeping channels of communications open and in 
constant use. 

TARGET OF EVALUATION OR MONITORING 

The Manhattan Children's Psychiatric Center is 
one of six State-run children's centers in New 
York State. It was officially opened in 1972. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

In addition to their responsibilities for moni- 
toring ongoing services, the MCPC board 
undertook to respond to staff complaints from 
numerous professionals and child care personnel 
regarding various aspects of the institution's 
functioning* Such concerns had been growing since 
the appointment o1 a new director about 2 years 
after the board was established. The airing of 
staff grievances was facilitated when, after 
continual urging by the board, notice of its 
meetings was posted (board of visitors meetings 
are open meetings, which are supposed to be 
posted and published in the community). 

Another issue arose early in the MCPC board's 
experience when members were asked by a 
concerned parent to conduct an investigation into 
an alleged instance of child abuse on one of the 
war do. The board was also asked to respond to 
allegations by a previous employee of misman- 
agement and incompetence in the administration 
of the center. 

TECHNIQUES USED 

Board members must be thoroughly familiar 
with their particular institution, and are therefore 
required to visit and inspect the facility at least 
twice a year — though, 5n fact, most members do 
this far more often. The board's assessment of 
conditions at the facilities and their recom- 
mendations are sent directly to the Governor and 
the State's Commission on the QuaUty of Care. In 
addition, boards have statutory powers to: 

• Investigate all charges against the director 
and all cases of alleged patient abuse of 
mistreatment made against any employee 

• Intervie* • patients and employees of the 
facility in pursuit of such investigations 

• Subpoena witnesses under oath, and require 
the production of any books and papers 
deemed relevant to the investigation 

The results of investigation, .elated to the 
facility or its management are reported directly 
to the Governor and the Commission. 
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An important part of the job of a board member 
is becoming familiar with all aspects of the 
facility, including the physical plant, the clients, 
and the staff. This necessitates frequent site 
visits, and establish'^i an accessible presence by 
talking to patients q..J staff. Though there may be 
an initial reaction to a board member as someone 
who is "snooping" or acting as a secret agent, 
board members should make clear thac the focus 
of their interest is the quality of care and treat- 
ment at the facility and that they welcome the 
assistance of staff, patients, relatives, and all 
others who can identify areas where improve- 
ments can be made as well as point out 
achievements for which the facility should be 
congratulated. 

The board at MCPC began its work by asking 
for various kinds of documents detailing the past 
history and present operation of the center, and 
received a wealth of material. However, this did 
not include a report by a former employee who 
had complairied of vaiiDus piactices at the 
facility which he felt were evidence of misman- 
agement and incompetence. Having heard about 
this document from outside sources, the president 
of the board requested a copy from the regional 
Office of Mental Health (OMH). Upon learning of 
this, the director of MCPC took great offense and 
accused the board of going behind her back, 
looking for negative material, and so forth. As a 
result, the board learned early in the process of 
the necessity for any "watchdog" group to be 
clear and direct about its activities, its channels 
for access to the materials required to carry out 
its work, and the necessity to use other channels 
available only after it is apparent that it is 
impossible to obtain such information directly and 
openly. 

FINDINGS OF EVALUATION 
OR MONITORING 

With regard to the issue of staff and adminis- 
trative conflicts, the board had to emphasize 
continually that their concern was the quality of 
care and treatment provided, and that they were 
not responsible for, nor capable of, running the 
facility. Thi board could act as a conduit for staff 
conca^ns, they could direct grievances toward 
appropriate channels (such ^s issues of evaluation 
and promotion, which were union concerns), they 
could investigate some staff concerns themselves 
(abuse of overtime, lack of structured foUowup to 
problems raised in committees, absenteeism, 
increased incidents, etc.), and they could report 
to the director of MCPC, the OMH and the 
Comnission on the Quality of Care on the results 
of their inquiry. 

In response to allegations of abuse, two 
members of the board with experience in the child 
cars field spent several days at the facility 
reviewing records and talking to staff, concluding 
that there had not been actual abuse, but that 
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staff had not been sufficiently trained in tech- 
niques for handling difficult residents. 

RECOMMENDATIONS 

Handling the issue of staff dissension and 
rebellion Is particularly touchy, and the board was 
extremely careful to stress their concurrence 
with staff concerns regarding patient care and the 
functioning of the institutions. However, the 
board emphasized that they were not in a position 
to take sides. 

On the abuse question, the board recommended 
that staff receive training In the handling of 
acting-out p?tlents, and noted that the line of 
responslolllty and authority on that ward had to 
be clarified. These recommendations were shared 
with the administration. 

STEPS TO ENSURE IMPLEMENTATION 

Continued monitoring by the board ensured that 
recommendations would be followed up. 

EXTENT OF IMPLEMENTATION 

Not applicable. 



SPECIAL BARRIERS OR SUPPORTS 

The major area where the MCPC Board has had 
difficulty Is in reaching oarents of those served ^^y 
the centf3r and establishing communication with 
them. In fact, though there have been some 
replacements on the board over Its almost 4 years 
of existence, an effort has been made to keep one 



of the seven slots open so that it may be filled by 
a parent or relative of a child cuirently at the 
institution. The effort has thus far been 
unsuccessful. 

A potential area of difficulty facing oversight 
bodies is the necessity to establish independence 
as a monitoring body along with the necessity to 
form a mutually constructive wc ^3 relationship 
with the administration of the facility. The 
possibilities of being "co-opted" by the director in 
order to avoid an adversarial stance, or of falling 
into a pattern in which confrontation is the 
primary mode of communication, both exist. It is 
not always easy to walk the line between these 
t' '0 patterns, and indeed confrontation may 
become necessary in certain situations. 

RESOURCES AND COSTS 

The board receives no salary. Its costs in 
connection with board duties are reimbursed. 

ADDITIONAL COMMENTS 

Citizen advocacy Is essential In our society 
with so many competing, although worthy, causes 
and with so many enormous bureaucratic struc- 
tures to negotiate and Influence. Advocacy for 
children Is especially important since traditionally 
the young have not had a strong constituency to 
press for better, more adequate, and appropriate 
services. In the arena of Institutional care of the 
mentally disabled youth of New York State, the 
boards of visitors have made and continue to 
make their voices heard on behalf of these young- 
sters, and are working together with others whose 
goals are similar— In the system and outside It. 



9. Advocacy Organization Sponsors Board and Care Survey 

David StSott 



Mental Health Advocacy Project of the Santa Clara 
County Bar Association Law Fomdatlon, Inc. 



SUMMARY 

In 19dl, the Mental Health Advocacy Project 
(MHAP) of Santa Clara County, California, 
sponsoiCd a survoy of all of the board and care 
facilities In the county serving mentally 111 and 
developmentally disabled adults. In orcier to 
collect Information about the hom*5s, MHAP 
recruited student and consumer volunteers to 
Interview board and care residents. The result of 
the survey wac the preparation of a guide titled 



For further information write David Schott. Hotising 
Specialljst, Mentil Health Advocacy Project. 711 East 
San Fernando Street, San Jose, California 95112, (408) 
294-9730. 
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Catalogue of Residential Care Facilities in Santa 
Clara County. In order to make the catalog 
accessible to a wlc!<) range of clients, symbols of 
various aspects of the i,?mes such as food, 
physical plant, and atmosphere were developed. 
The biggest problem encountered by MHAP was 
gaining access to board and care homes because 
of the resistance of the operators. Over all, the 
project showed that board and care residents can 
make judgments about their circumstances and 
can use In.'ormatlon to make decisions about their 
care. 

TYPE OF ORGANIZATION 
The Mental Health Advocacy Project was 
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organized in 3uly 1978 under the sponsorship of 
the Santa Clara County Bar Association Law 
Foundation. Initial funding for the project was 
provided by a grant from the American Bar 
Association Commission on the Mentally Disabled. 
MHAP focused its activities on the reform of the 
mental disability system through direct case 
representation of clients, as well as impact 
litigation on behalf of mentally ill and develop- 
mentally disabled clients. The project, which Is 
located In the center of San Jose's board and care 
community, has expanded Its program and now 
receives funding from multiple sources Including a 
contract with Santa Clara County to provide 
advocacy services. The project outlined In this 
case was supported by a $25,000 program 
development grant from the State Department of 
Developmental Services. 

EVALUATORS OR MONITORS 

'i ne assessment of board and care facilities was 
undertaken by an Independent team that Included 
MHAP staff, ten undergraduate social work 
students from San 3ose State University, and 
three consumers of board and care services. In 
order to recruit the social work students, f/)HAP 
staff made a presentation to a class detailing why 
the students should fulfill their community 
Involvement requirement by working with tne 
advocacy project. The consumers were recruited 
by word of mouth among the board and care 
residents In the vicinity. Consumers' rsactions 
were also solicited by questionnaire. 

REASONS FOR THE EVALUATION 
OR MONITORI^ 

Prior to th , few sources of Informa- 

tion regardlnc md care facilities existed In 

the county, arc . a that did—such as a refer- 
ence guide prepared by the Residential Care 
Association— were written by, and for profes- 
sionals or providers. MHAP staff felt that it was 
Important that the client perspective on the 
adequacy of board and care facilities be solicited. 
In order to support this activity, MHAP staff 
applied for a grant from the State to help board 
and care residents to become more active as 
consumers of service. The notion of compiling 
client assessments of facilities Into a catalogue 
did not emerge until the project was underway. 

LEVEL OF PARTICIPATION 

MHAP staff were involved In the evaluation at 
all stages of the process. They recruited the 
students and consumers, participated In the design 
of the beard and care survey, onducted training 
sessions, assigned homes to surveyors, maintained 
periodic contact with the volunteers, met with a 
.arlety cf Interest groups to facilitate access and 
to ease political problems, assisted In the ratings. 



and wrote up the resultc. Consumers and students 
conducted the onslte surveys and contributed to 
the final ratings. 

TARGET OF EVALUATION OR MONITORING 

The objective of the <^urvey was to derive 
ratings on all board and care facilities serving 
mentally ill and developmentally disabled persons 
in Santa Clara County. Homes serving alcoholics, 
drug abusers, senior citizens, or children were 
excluded from the assessment. Homes surveyed 
varied from 2 residents to 45-50, though the 
majority hrused between 15 and 25 persons. 
Approximately 80 percent of the hemes assessed 
were for-profit organizations. Of the cotal 
number of homes targeted, surveyors were able to 
develop ratings on over 75 percent. In order to 
maintain the credibility of the survey, staff 
excluded homes frc ti the ratings where the 
surveyors were unable to contact a large enough 
sample of residents. In many Instances, Inade- 
quate sample size was the direct result of an 
mablllty to gain access to the home. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The board and care assessment focused on five 
aspects: foo^, the quality of the physical plant, 
and atmosphere of the home (i.e., the extent to 
which It was homelike or Institutional), the 
fairness of the house rules, and the extent to 
which residents were encouraged to be 
Independent. 

TECHNIQUES USED 

The first step In the assessment was the design 
of a questionnaire. Once a draft had been 
prepared, MHAP staff conducted a field test using 
a group of consumer? of whom roughly one-third 
were current board and cara r'^sldents, one-third 
were former residents, and one-third had never 
been In board and care facilities. Based on the 
results of the field test, revisions were made to 
the questionnaire. The questionnaire was struc- 
tured to elicit short answers, but residents were 
also given the opportunity for open-ended 
responses. The next step Involved training the 
students and consumers In administering the 
questionnaire and also providing an orientation to 
board and care facilities. FoUowup meetings were 
held with the surveyors to clarify any questions 
and problems regarding the questionnaire. 

Incidentally, Interviews with board and care 
residents were conducted by teams made up of 
students and consumers. Eventually, consumers 
were able to conduct interviews on their own. By 
and larne, residents were Interviewed onslte at 
timos when they were likely to be at the home 
(e.g., around dinner time). Whenever possible, 
reside! <ts were Interviewed privately, although 
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this was not always possible. If the operator would 
not grant access to the interviewers, they 
canvassed day programs to identify residents of 
particular homes. Gaining access to some board 
and care facilities proved to be the biggest 
problem at this stage of the assessment. 

Upon completion of the survey, MHAP formed 
three-person teams to develop the ratings for 
each facility. Teams developed consensus 
statements for each facility (e.g., if 8 out of 10 
residents said the food was very good, then the 
consensus statement read, ''most of the residents 
say the food is very good**). The result of this 
process was a one-page digest of responses per 
facility that reflected the average rating of the 
residents or the consensus of those contacted. At 
the top of each page, the operator of the home 
was given an opportunity to describe his or her 
facility. MHAP also offered the operators an 
opportunity to prepare a rebuttal to resident 
ratings and to have the rebuttal included in the 
catalogue. However, none of the operators took 
advantage of the opportunity. 

In order to make the guide as accessible as 
possible, MHAP staff spent time designing and 
field testing symbols that could be understood by 
semiliterate clients. For instance, the symbol for 
food was a plate and silverwaie and the level of 
the rating was indicated by one to four stars 
underneath the food symbol. 

FINDINGS OF EVALUATION 
OR MONITORING 

By and large, board and care residents tended 
to rate their facilities fairly highly. This is not 
surprising, given the fact that research has 
consistently shown that mental health clients tend 
to express satisfaction with services received. In 
those few instances where residents made highly 
critical comments, MHAP staff required that 
some other evidence be available before the 
information was included in the catalogue. 

RECOMMENDATIONS 

Based on the experience gained during the 
survey, MHAP staff are convinced that board and 
care residents can participate in decisions 
regarding their surroundings. They neither lack 
the judgment, nor— as some family members 
maintained — are they harmed by taking respon- 
sibility for decisions about their place of 
residence* Further, given the problems that some 
surveyors had in gaining entry, project staff 
strongly support provisions that guarantee free 
and open access to board and care facilities and 
residents. 

STEPS TO ENSURE IMPLEMENTATION 

The completed guide, titled Catalogue of 
Residential Care Facilities in Santa Clara County, 
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was advertised to several newsletters and copies 
were made available in the MHAP office for 
persons who wished to examine it. 



EXTENT UF IMPLEMEN I ATION 

To date, several consumers have come to the 
MHAP office to consult the guide, but only five 
copies have been distributed. In part, the limited 
circulation is attributable to a shortage of funds 
for reproduction and also to thie ^*act that some of 
the material is now dated. Requests for informa- 
tion regarding the guide and the assessment 
process have been received from around the 
State, but it is not clear whether similar assess- 
ments have been conducted as a re^^ult. MHAP 
staff have not advertised the availability of the 
guide among county casework staff on the 
assumption that they have their own sources of 
information. 



SPECIAL BARRIERS OR SUPPORTS 

The major barrier to the board and care 
assessment was securing access to board and care 
residents. Operators of the homes were, in some 
instances, extrenr.vily reluctant to let surveyors 
into the homes since they were fearful that the 
results of the survey might damage their reputa- 
tion. In order to overcome the resistance of the 
operators, MHAP staff met with the providers on 
numerous occasions. Ultimately, advocacy staff 
worked with the State licensing agency and others 
to secure aw interpretation of the law that 
allowed access to board and care residents if the 
residents warted to talk to the surveyors. 
Eventually, surveyors were able to gain access to 
most homes, and where they were unsuccessful, 
they sought out residents in day programs. 

Another group that expressed disagreement 
with the aims of the project was a vocal minority 
of the local parents organization. These parents 
argued that mentally ill and developmentaliy 
disabled persons should not be permitted to make 
judgments r^gardinr where they should live 
without the supervibiun of mental healH profes- 
sionals or family members. 

Groups in the county concerned about the 
project, including board and care operators, 
providers, parents, and the county board anH care 
consulting unit, approached MHAP and suggested 
that they form an advisory body* MHAP agreed 
and met with the groups at least sc^ven times 
during the course of the assessment. At the 
beginning, the advisory group wanted more 
control over decisions regarding the survey but 
MHAP staff ultimately maintained control over 
the project. 

On the positive side, MHAP staff point to the 
consumers advisory group as a supportive body 
during the conduct of the survey. This group was 
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autonomous since t*i8 consumers were worried 
that their 'Hewpoints would be overidden in the 
professional advisory group. MHAP staff also 
noted that the county mental health association 
was supportive of the project. 



RESOURCES AND COST S 

The project required the time of approximately 
one full-time MHAP staff nr.ember for 1 year. 
After that time, MHAP staff were i ivolved 
periodically until the document wzs completed. 
Volunteers spent 10 hours in training, 30 hours in 
interviewing, and another 10 hours developing 
consensus statements and tallying data. The grant 
for the project was $25,000. The project took 
approximately 18 months from beginning to end. 



ADDITIONAL COMMENTS 

If such a project were to be conducted again, 
MHAP staff suggest that funds for publishing the 
catalog be built in from the beginning, and that 
problems of -jcess be dealt with from the start. 
They also suggest that the advisory committees 
be formed at the project's inception. With respect 
to interviewing, it is suggested that wherever 
possible clients be interviewed privately. This 
point also suggests that more attention be paid 
generally to the circumstances under which 
clients are interviewed in order to garner the 
rCiOSt responsive results. Finally, attention should 
be paid to ensure that the information is cun ^»nt 
when the guide is distributed, and that an 
inexpensive method for broad distribution be 
devised. 



10. Patients Group Evaluates Care 

Gabriel Manasse in consultation with Charles Gold 

BrentYfood Veterans Administration Medical Centpr, Los Angeles, Calif omia 



SUMMARY 

The Patients Advisory Consultation Team 
(PACT), organized in 1973, is a volunteer group 
composed of patients at the Brentwood Vetera 's 
Administration (VA) Medical Center who provide 
feedback from the consumer's point of view. 
PACT'S functions include preparation of an annual 
report for the director of the facility citing the 
strengths and weaknesses of the institution; and 
representation on the clinical and administrative 
executive boards; and membership on the dean's 
committee— an executive group linking the 
medical center with the University of Cclifornia 
at Los Angelas. PACT also serves as a primary 
recipient of complaints and investigates such 
complaints for the medical center In the summer 
of 1981, PACT also organized a national confer- 
ence focused on "Patient/Staff Cooperation in 
Psychiatric Care." Major concerns of the gioup 
over time have 'ncluded discrimination against 
mentally ill pei ns, inadequate housing in the 
community, shortage of rehabilitation services, 
and individual client Ights. 

TYPE OF ORGANIZATION 

PACT is a group of patient volunteers. It is 
open to any patient at Brentwood VA Medical 
Center whether inpatient, outpatient, or even 



For further information write Charles Gold, President, 
Patients Advisory and Consultation Team, VA Medical 
Center, T38, Los Angeles, CA 90073, (213) 478-3711 
(ext. 2151). 



those who have ceased formal therapeutic 
relationships. One staff person is currently a 
regular member, and staff have often served as 
consultants. Since its inception, this group has 
been part of the administrative struct^ re of the 
Brentwood VA Hospital. Inclusion in that struc- 
ture, with direct links to the director of the 
center and the education service, were considered 
crucial by the volunte'^r patients who proposed 
the establishment of PACT. 

EVALUATORS OR MONITORS 

Regular members of PACT are all active or 
former psychiatric patients. Their backgrounds 
are quite varied. The current pre.iident of PACT 
was a carpenter prior to his physx-^l, and later, 
psychiatric impairment. Others have been 
merchants, workers in v^irious industries, 
teachers, etc. Tasks are assigned by the group in 
accordance with its assej^sment of the volunteer's 
capabilities. 

REASONS FOR THE EVALUATION OR 
MONITORING 

The evoiiition of the PACT idea was rather 
serendipitous. Initially, a small group of volunteer 
patients was assembled in an attempt to get their 
suggestions for the development of a new acute 
treatment program. It soon became evident tha*, 
the volunteers had little to offer to the desigi i Df 
a new treatment program per se, but were able to 
identify major institutional issues that had 
traditiuiiaUy remained unaddressed or insuffi- 



37 

C6 



ciently addressed. Including client perceptions as 
an ongoing port of institutional assessments 
seemed higly dobirable. The hospital director was 
petitioned to establish the group with those 
purposes in mind. The ability of this group to 
evaluate patient complaints was discovered and 
led to the creation of the Patient Staff Treatnient 
Review Board (PSTRB). 

LEVEL OF PARTICIPATION 

The level of involvement of PACT varies 
according to the issue involved. PACT submits an 
annual list of recommendations to the director 
which is based on the perceptions of the collec- 
tive group over the year. The list notes the 
strengths and weaknesses in the functioning of the 
medical center. Various members sit on policy 
groups and provide ongoing consultation. More- 
over, members are frequently appointed to task 
forces in order to investigate particular issues. 

Patient complajnts are also brought to the 
group. The type o' complaint determines the steps 
that will be followed. Frequently the organization 
assigns a member to look into a situation and give 
a report. Where it is appropriate, hospital 
authorities may be involved. 

TARGET OF EVALUATION OR MONITORING 

Brentwood VA Medical Center is a psychiatric 
institution providing care to veteran pcMents in 
the Los Angeles area. The fundamental r?cus of 
PACT'S attention has been on this particular 
institution and its programs. Late in 1981, this 
institution was merged administratively with the 
medicine/surgery sister institution, Wasworth 
Medical Center. Negotiations are currently 
underway to define the relationship of PACT to 
this new entity. 

PROBLEM'S OR ISSUES EVALUATED 
OR MONITORED 

Over the course of the years, many different 
types of issues have been evaluated by PACT. 
Problems in the administration of support services 
have been a frequent focus. For instance, 
problems in checking in or retrieving clothes from 
the personal effects section formed the basis for 
one major project. Obtaining placement of a pay 
telephone in an area accessible to a sometimes 
locked ward was another project. Additionally, 
PACT has also been active in increasing public 
awareness of ii.e difficulties of finding adequate 
housing, supporting patients and trying to deal 
with discrimination against psychiatric patients in 
the community, helping patients become aware of 
benefits to which tney are entitled, and reviewing 
patient complaints. The latter can range from 
disputes about treatment (most commonly 
disputes about discharge planning) to questions of 
privacy and even cases of alleged patient abuse by 

38 



staff. An ongoing issue is the need to divert more 
of the center's resources into rehabilitation 
effort's rather than acute treatment. There has 
been less focus on the physical olent. At times 
apparent institutional cJeficiencies have been 
worked on directly by the group. For instance, at 
one time PACT ran a nightly social club for 
inpatients and outpatients. 

The focus of the patient/staff conference held 
m June 1981 was on the limits of treatment 
imposed by the state-of-the-art in psychic.try and 
its traditional deemphasis of patients concerns. 
This issue was addressed from the perspective of 
the chronically mentally ill. Since the conference 
was generated by a VA patient group, much of the 
emphasis was on VA programs. 

TECHNIQUES USED 

Generally peaking, problems are brought to the 
attention of ?ACT through first hand reports 
from persons involved. Patients who report 
inadequate or incorrect treatment can demand 
that the PSTRB be convened. This group, 
consisting of four senior staff and three patients, 
reviews disputed treatment decisions and has the 
power, with the concurrence of the hospital 
director, to overrule the treatment team's 
decision. As a rule, an investigation is conducted 
either through interviews with other parties, 
or— if there is a dispute—by gathering 
information on a particular situation. For the first 
8 years, nneetings occurred twice weekly. Now the 
group conducts one business meeting weekly. The 
other meeting time has been devoted to a new 
project, holding orientation sessions for first-time 
admissions to the medical center. 

The most important activity of :he team is the 
maintenance of accessibility. PACT can be 
contacted by telephone or a visit to their offices 
during normal working hours. In some instances, 
for particular issues, PACT has utilized surveys. 
For instances, a survey on the adequacy of 
services to female veterans was prepare^ and 
conducted by PACT in conjunction with ^caff. 

In order to dramatize client concerns, PACT 
organized a national conference titled "Patient/ 
Staff Cooperation in Psychiatric Care" in 1981. 
Patients, prominent VA staff from around the 
country, and community leaders met for 2 days to 
discuss policy questions developed by PACT. 

FINDINGS OF EVALUATION OR MONITORING 

There is no simple way to define findings that 
span 9 years. Speaking generally, there has been a 
sense that intervention focuses more vigorously 
on acute care than on chronic care and 
rehabilitation. (This is not perceived as 
institution -specific, but as characteristic of the 
field in general.) Clients feel that care of the 
residua) illness and rehabilitation ought to be 
emphasized. Clearly there are many social issues 
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that confound treatment: inadequate housing, dis- 
crimination in the community, poor public 
transportation, etc. These make all the attendant 
problems of mental illness much more .severe. 

On the other hand the group is greatly pleased 
by the support it has found in the administration 
of the medical center and the medical center, in 
turn, has been generally pleased with the work of 
the group. The interactive dialog between the 
staff and patients is perceived as healthy from 
both points of view. 

RECOiVBS/IENDATlONS 

The general trend in recommendations has been 
toward Increased emphasis on rehabilitation and 
care for the chronically ill as opposed to acute 
care. There also has been a strong wish to 
increase attention to the physical needs of th^> 
psychiatric patient, a desire which may be 
partially realized through the current merging of 
facilities. 

A major recommendatiori of the group has been 
that other institutions encourage similar patient 
groups. Many problems have been resolved by this 
group in a way that is believed to be highly 
^referable to traditional means. 

STE PS TO ENSURE IMPLEMENTATION 

Generally the relationship with the medical 
center has been cooperative. There is little that 
the group can do to enforce its recommendations 
but they havt had the support of the 
administration in an overall sense and most 
recommendations have been influential, even if 
not wholly adopted. The only way to further 
ensure adoption is to monitor implementation 
o,^r time. This has been done over the course of 
9 years. 

EXTENT OF IMPLEMENTATION 

The general stance of the administration 
towards psychiatric patients has greatly improved 
over the course of 9 years. A new awareness of 
the needs of the chronically mentally ill has 
seemingly emerged and inclusion of patients in 
many policymaking bodies gives testimony to the 



institution's respect for client views. Of course, 
the PACT voice is only one of many, and their 
perspectives have not boen entirely adopted. The 
working relationship, however, remains sound. 

SPECIAL BARR IER S OR SUPPORTS 

A constraint, which is inevitable in this type of 
arrangement, is that many of the patients are in 
active treatment with members of the staff. 
Clearly this leaves them partially restrained in 
their critical evaluations. Nonetheless, by forming 
this type of group, residents have provided 
support for one another, and judging by reaction 
of the administration, the evaluations on the 
whole have been honest and open. The major 
support for the activity is provided both by PACT 
and the hospital administration. 

RESOURCES AND COSTS 

There are no paid staff who have partici. ^ated 
in this activity in u regular fas*^ion. A small 
amount of money has been allocated for supplies 
although most supplies have come through 
voluntary donations. The hospital provides some 
housekeeping services, has allocated a snf>all 
unused building to PACT, and allowed a telephone 
to be installed for the group's use. A small band 
of dedicated patient volunteers contributed 
enormous amounts of time over the last 9 years to 
ensure the group's continued functioning. 
Resources for the conference hosted last year 
came mostly through donations; a small amount 
was supplied by units of the VA, some services 
were donated by the University of California, Los 
Angeles, and many persons attended on their own 
time. 

ADDITIONAL COMMENTS 

The PACT group remains viaole and valuable. 
However, there are chronic problems in recruiting 
new persons to take on the burdensome and 
time-consuming tasks. Conceivably, it could be 
desirable to pay some of the patient volunteers 
for their time, although difficulties in arranging 
stipends might outweigh the benefits that could 
be achieved. 
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How Much Win You Be Involved? 



After Doug and his committee have discussed the mental health 
association's organizational strengths and limitations, the conversation 
turns logically to a discussion of the level of involvement that the evalua- 
tion committee should have in any information collection and planning 
activity. 

Since Doug has already secured an agreement from the comm'.ctee 
that they will take ?. faiily acti.o role in the evaluation project selected, 
he suggests that tliey have therefore ruled out serving as citizen responders 
to someone else's evaluation. The question, then, is whether the committee 
will conduct the study independently, work in partnership with the county, 
or bring in an outside consultant. 

Though the association has a good working relationship with tiies 
county department of mental health and the county advisory board, the 
interests and priorities jf these two groups differ, at least for the time 
being, from the interests of the mental health association. The advisory 
board is in the midst of assessing prevention programs in the county, and 
the county department personnel are currently going through a major re- 
organization. Further, a collaborative evaluation would to some extent 
hamper the committee's ability to be critical of county planning prac- 
tices—an issue that the mental health association feels strongly about. 

The factors in favor of conducting an independent evaluation include 
the assocr-tion's reputation and past accomplishments. Two years ago, the 
association conducted an independent evaluation of children's services that 
resulted in major reforms at the county level and that became a model 
statewide. During the course of the evaluation, the association garnered 
significant respect from the mental health and generic human services 
agencies in the county. An independent evaluation would also mean that 
the committee could collect information that would be useful to the in- 
ternal needs of the association as well as to the system as a whole. Finally, 
since there are some tensions between local county staff and State staff, 
maintaining independence from the county would free Doug and his com- 
mittee from any unnecessary political baggage. 

The committee decides to work independently for the reasons dis- 
cussed above, and they now have to determine to use an outside consultant 
to assist with the project. Two factors influence this decision. First, the 
association, like many voluntary organizations, does not have any discre- 
tionary funds and would have to seek foundation support to pay a consul- 
tant. Second, Doug feels strongly that the association volunteers should 
conduct the study in order to become more familiar with the system. The 
committee therefore decides to conduct an independent evaluation. 



The cases in the following chapter cover vari- 
ous levels of Involvement of citizens and con- 
sumers In evaluating and monitoring mental 
health and related services. The spectrum ranges 
from cltizeti participation as respondents to key 
informant surveys to totally independent moni- 
toring and evaluation projects conducted by 
citizens. 

Subjects of Evaluation 
In case #11, Donald Elb describes a joint ven- 



ture sponsored by a mental health center, a health 
systems agency, and a State university. The 
evaluation, which was aimed at the development 
of a long-range cental health plan, utilized per- 
sonal interviews with local citizens as the core of 
the needs assessment portion of the project. 
Though citizens were not involved in the design of 
^he project, their views were considered critical 
. che groups involved. 

In the next case, written by Suzan Wolpow, 
consumers are used as tixe subjects of evaluation, 
although in this instance they play a more active 
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role than the citizens in the above case. The 
consumers are clients of a psychosocial rehabil- 
itation club dnd are participants in an agencywide 
evaluation. They in fact helped to structure the 
evaluation questions and also reacted to the re- 
sults. Their participation was critical given the 
consumer-oriented nature of the agency. 

Solicited Reactors 

In case #13, written by Lois Burgner, citizens 
are used primarily as reactors to evaluation ma- 
terials prepared by a mental health center. The 
citizen/consumer team, called a citizen review 
group, reviewed agency materials, evaluation re- 
sults, and other relevant documents in order to 
develop recommendations for center adminis- 
trators. The review group included consumers of 
services and their families and representatives of 
agencies in the community that utilize center 
services. According to the author, the citizens 
and agency representatives made practical and 
responsive recommendations and all concerned 
seemed pleased with the process. Though the use 
of citizens and consumers as reviewers of evalu- 
ation results is certainly beneficial, this case 
shows two of the possible weaknesses of this ap- 
proach — diminished consumer interest and in- 
volvement over time, and a lack of commitment 
on the part of the center board and adminis- 
tration, at least initially, to cany out the recom- 
mendations of the review group. 

Partners in Evaluation or Monitoring 

The subsequent three cases describe citizen 
evaluations conducted in partnership with public 
mental health groups. Case #14, written by 
Barbara Geddie, describes a cooperative ar- 
rangement between a State mental health asso- 
ciation and a State department of mental health. 
The arrangement involved the placement of an 
association member on each site-visit team con- 
ducting reviews of service monitoring around the 
State. Ms. Geddie points out the benefits of 
having citizens serve on site-review teams, in- 
ducing the fact that citizens brinf a certain 
sensitivity to client-related and community- 
related issues and that citizen team members are 
able to advocate for program expansion in ways 
that other team members are not. Further, in- 
formation gained by citizens during the review 
process is valuable to the mental health associ- 
ation in designing its lobbying strategies. 

The next related case, prepared by Diane Rich, 
involves a multiple partnerdiip — a State mental 
health association, the State mental health agen- 
cy, and the Federal regional ADAMHA office. 
Tliis activity also involved site visits to mental 
health providers around the State in a review 
process that integrated both State and Federal 
requirements. In this project. State and Federal 
officials had to be con^rtnced of the value of cit- 



izen participation, but as time went on these 
same officials became reliant on the citizens for 
the perspective they brought to the monitoring 
activity. 

In the final partnership case, Jean Fiore de- 
scribes a joint review of a coua7ty's emergency 
services conducted by the county advisory board 
in conjunction with county mental health and 
human services agencies. The partnership— which 
included the provision of technical assistance and 
other in-^kind support — proved beneficial to the 
citizen group since the development of survey 
instruments turned out to be a very complex task. 
The partnership also tended to ensure an audience 
for the results. 



Independent Evaluators of Monitors 

All three of the next cases involve citizen or 
consumer groups that conducted or are conducting 
independent eveliiation or monitoring activities. 
The first example, written by Ellen Colom Deacon 
and Thomas M. Quilter, reflects the joint efforts 
of a county mental health association and con- 
sumers of services from the area. The study was 
initiated by the association in order to gain cit- 
izen assistance in the development of an advocacy 
sjrstem and to develop a better understanding of 
the consumer view of mental health needs in the 
area. Using the nominal group technique, facil- 
itators worked with a group of mental health 
consumers to identify priorities for service im- 
provement. The association also conducted in- 
terviews with current and former consumers. By 
conducting an external or independent assess- 
ment, the association was able to collect infor- 
mation that was both useful for its own internal 
planning purposes and also for its larger system 
reform agenda. 

Case #18, written by Rita T. Parle, involves an 
ad hoc citizens body appointed by a mental health 
board of trustees, but which functioned in a to- 
tally independent fashion. The study committee 
was created as a result of controversy surrounding 
the management of the area's mental health 
center. Given the nature of the problems un- 
covered by the committee, vrtiich included con- 
ficts of interest, previous mismanagement, and 
nepotism, it is hard to imagine that a group 
wor'ung in partnership with the agency could have 
had the freedom to explore these problems. 

The last case in this category, written by 
Christine Anderson, Michele BoUenbeck, and Iris 
Freeman, describes the activities of an indepen- 
dent citizen advocate group working vdth elderly 
persons in nursing homes. The case poignantly 
portrays the ways in which individuals can have an 
impact on the lives of vulnerable persons at risk 
of abuse or neglect. It once again emphasizes the 
importance of routine monitoring of ntarsing home 
facilities — especially for the consultation of in- 
dividuals with no other family or friends. 
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External Consultant 

The final case in the chapter describes a col- 
laborative evaluation effort that involved a pro- 
fessional evaluator and volunteer evaluation team 



members. The case clearly indicates the ways in 
which volunteers — in this instance, professional 
voiunteers— can augment the resources of an ex- 
ternal consultant and also provide hixr or her with 
important insights and specialized expertise. 



i1. Citizens as Subjects of Evaluation 

Donald J. Eib 

Newaygo County Merited Health Center 



SUMMARY 

The board of directors of the Newaygo County 
Mental Health Center (NCMHC), as part of the 
developnnent of a planning system, u'^dertook a 
three-part project: (1) a basic study for long- 
range planning; (2) the development of a 
long-range plan from an internal perspective; and 
(3) conduct of a needs assessment through 
interviews with knowledgeable persons. The 
project addressed both internal and external views 
of the agency, and also provided linkages between 
a university, a health system planning agency, and 
the center. 

TYPE OF ORGANIZATION 

The Newaygo County Mental Health Board is 
comprised of residents of the county appointed by 
the Newaygo County Board of Commissioners. 
The board sets policy for the operation and 
delivery of services to the mentally ill and 
developmentally disabled residents of the county. 

EVALUATORS OR MONITORS 

This study was conducted by a student intern in 
policy and planning, as a second year M.S.W. 
placement. The agency provided a stipend and 
summer employment to the student. This 
individual developed a design for the study in 
association with planning staff from the regional 
health systems agency (HSA). 

REASONS FOR THE EVALUATION 
OR MONITORING 

The Newaygo Count;, * lental Health Center was 
established in 1972. Several attempts at devel- 
oping a comprehensive planning system have been 
made since that time. The NCMHC subscribed to 
the concept of the "^balanced service system** 
standards of the Joint Commission on Accredi- 
tation of Hospitals (3CAH 1979). The 3CAH 



For further information v/rite Donald J. Eib, A.C.S.W., 
Director, Newaygo County Menul Health Center, 1049 
Newell Street, P.O. Box 8367, White Cloud, MI 49349, 
(616) 689-7330. 
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standards address issL^o of planning and consumer 
involvement. This concept, along with a long- 
term need of the board for a well-organized 
planning effort, brought the project into focus. 
Incidentally, the university's school of social work 
needed a student placement, and the regional 
HSA, which has several rural community mental 
health programs in its service area, needed a 
model plan. The convergence of these three needs 
meant that all agencies could benefit from a 
cooperative effort. 

LEVEL OF PARTICIPATION 

A? noted above, the evaluation system was a 
cooperative effort among three agencies. The 
entire staff of the center participated in the 
long-range planning, as well as providing inform- 
ation to the individual (student) who conducted 
the interviews with key informants (knowledge- 
able persons)^ Citizens were the subjects of 
evaluation since their views of service adequacy 
were at the heart of the study. 

TARGET OF EV AL UATION OR MONITORING 

This project focused on the total agency. The 
Newaygo County Center is the sole service 
provider of mental health and developmental 
disabilities services in a rural county with a 
population of only 34,000. The poverty level is 
high and the unem ploy men c U;vel in good 
economic issues is in excess of 10 percent, and is 
currently over 25 percent. The agency provides a 
full range of CMHC services including outpatient, 
partial hospitalization, cace management, 24-hour 
emergency services, prevention, family crisis 
services, and day programs. The center is also 
licensed to provide outpatient substance abuse 
services. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The survey of citizen concerns posed the 
following questions: 

• What kind of mental health services does 
your community need? 
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• What kind of mental health services might 
your neighbors need? 

• What kind of mental health services might 
your family/relatives need? 

• Describe the services currently available 
from the Newaygo Cour.'-y Mental Health 
Center. 

• Kow would you like the services delivered? 

TECHNIQUES USED 

Forty key informants were selected and 
Interviewed. Because the center was just starting 
to gather planning information, it was considered 
best to proceed in a very general manner, 
focusing on a broad area of citizen concern which 
could be expanded in later assessment efforts. 
Key informants were selected on the basis of area 
of residence, sex, religious affiliation, age, race, 
economic status, length of residency in the 
countyt and national origin. One individual 
conducted all of the Interviews, most of which 
were held in the informant's natural setting (i.e., 
workplace, home, etc.). It was felt that the 
combination of a personal interview, with open 
ended questions, maximized the latitude of 
responses. 

FINDINGS OF EVALUATION OR MONITORING 

Two major areas were found to be concerns in 
the key informant responses. These included the 
agency*s need to increase and expand public 
education and information efforts, service 
delivery, and the possibility of establishing agency 
branch offices. 

RECOMMENDATIONS 

One recommendation was that the center 
should Improve communications with the com- 
munity through Increased efforts in public 
Information and education. 

A second recommendation supported the 
expansion of the center through multiple sites and 
modes of delivery. Specifically, satellites and 
additional group counseling services were 
suggested. Respondents persistently commented 
on the need to reach more people with the 
resources available. 

STEPS TO ENSURE IMPLEMENTATION 
The recommendations were made a part of the 



NCMHC's long-range plan which is reviewed on a 
regular basis by the agency's administration, the 
program cominittee of the board, and, finally, the 
full board, T'..j extent of achievement of long- 
range goals in the plan is published for internal 
and external review every year. 

EXTENT OF IMPLEMENTATION 

On the basis of this study, public information 
activities have increased by 50 percent. Because 
of limited funds, implementation of the sugges- 
tion that services be expanded on a decpntralized 
basis has not been fully implements . However, 
services provided to agencies such as public 
schools have changed in orientation from 
consultative to more direct service. 

SPECIAL BARRIERS OR SUPPOR TS 

The process of development and implementa- 
tion of the study met with limited resistance. 
There was a remarkably high level of response 
from key informants, with only one respondent 
refusing to participate in the study. The study 
received full support from the staff, board, and 
administration 

RESOURCES AND COSTS 

NCMHC is a small agency, serving a rural 
catchment area. Fiscal resources are limited, as 
well as expertise for the development and 
implementation of such an evaluation. As a result, 
the resources of a university, the HSA, and the 
center were combined to provide needed infor- 
mation to each of the participants. The total cost 
for the project was $9,000, which included a 
stipend for the student study director, a project 
design by university staff preceding the initiation 
of the project, travel, and other costs. 

ADDITIONAL COMMENTS 

By utilizing the resources of the HSA, the 
university, and the available fund of the center, 
the projdct was able to meet the needs of all 
cooperating organizations. The HSA has a model 
plan for distribution to other mental health boards 
in their region, the university secured a place- 
ment appropriate to the educational needs of the 
school of social work, and the NCMHC Board has 
as a final product a plan that integrates both 
internal and external concerns. 
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12. Consumers Offer Feedback on Social Club Functions 

Suzan Woipow 



Center Club, boston, Massachusetts 



SUMMARY 

In the Spring of 1982, an evaluation of Center 
Club, a psychosocial rehabilitation club for 
former psychiatric patients, was conducted. The 
purpose of the project was to evaluate club 
member satisfaction with Center Club services 
and program components, as well as to identify 
club member needs. Evaluation questions were 
generated through a series of discussions involving 
the evaluator, club members, and staff. The 
questions elicited information on the overall 
functioning, needs, and goals of Center Club. The 
survey was administered by members and staff to 
all club members attending the program during a 
10-day period. Results were compliled and 
reported by the evaluator and are presently being 
reviewed for implementation by a committee of 
six club members and one staff memLer. 

TYPE OF ORGANIZATION 

Center club is a 23-year-old psych'^cocial club 
with a priinary focus of meeting the social 
support needs of former psychiatric patients in 
the community. The club is part of Center House, 
Inc., a comprehensive agency that offers multiple 
services to former psychiatric clients. It is a 
non treatment setting with no psychiatrists or 
psychologists on staff. Rather than being placed 
in a patient role, club members join in a social 
club setting where they can participate at 
whatever level they choose Center Club operates 
7 days a week on the third floor of a ••Y*' in 
downtown Boston. It offers a wide variety of both 
regular and fluctuating programs. The center's 
pre vocational program includes components of 
maintenance and housekeeping, clerical and 
communication, thrift shop, and food service. 
Recently, the prevocational program increased in 
size and variety, and payment to members for 
work in ♦'^^e food service and housekeeping unit 
was elin'iinated. 

Attendance at Center Club is completely 
voluntary and there is an active membership of 
approximately 210 people who attend the club at 
least once a month. The council of officers is 



For further information yrnie Suzan Woipow. 46 
Carlton Street, #5, Brookline. MA 02146, (617) 
277-1529. Ms. Woipow is a former s'aff member at 
Center House, Inc. and served as the independent 
evaluator on the project. 



ERLC 



elected by club members. Council meetings are 
held weekly and are open to all club members. 
Staff are available to aid in coordination of all 
aspects of the program, to encourage participa- 
tion, to develop and encourage relationships with 
and among members, to aid in the social growtn 
of members, and to help in handling any crises. In 
the prevocational program, staff are available to 
coordinate and supervise, and to provide coun- 
seling and referral services. 

EVALUATORS OR MONITORS 

The primary evaluator was a former staff 
member from Center Club who was receiving 
training in program evaluation at the time of the 
survey. Dr. LeRoy Spaniol, director of research at 
Boston University's Center for Rehabilitation 
Research and Training in Mental Health, provided 
assistance throughout the project. Center staff 
and club members generated most of the content 
in the survey instrument. In addition, staff and 
club members monitored the administration of the 
survey. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The Center Club needed some form of evalua- 
tion and data collection in order to provide 
program information and to involve club members 
in policy and decisionmaking. It had been 6 years 
since the last large-scale, formalized evaluation 
activity. In the interim, sporadic unsystematic 
surveys were carried out to collect information 
about members* program preterences. 

LEVEL OF PARTICIPATION 

Evaluation questions were formulated through a 
series of meetings with staff ^nd club members. 
The survey instrument was draf Led by the outside 
evaluator and reviewed by club and staff mem- 
bers, resulting in several revisions. The final copy 
of the survey was typed up for dis^ribution by a 
member of the clerical and conrrr.uiucations work 
cluster. Club and staff members were also 
primarily responsible for the actual administra- 
tion of the questionnaire. Many members were 
excited by the prospect of involvement as ran be 
seen by the response. Over a period of IC days, 99 
out of 210 club members completed lengthy 
surveys, an extremely high number for such a 
short period. 
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TARGET OF EVALUATION OR MONITORING 

The evaluation focused on the Center Club 
program as a whole, including socialization and 
prevocational activities. 

PROBLEMS OR ISSUES EVALUATED 
OR MOinIITORED 

Some of the major evaluation questions were as 
follows: 

1. Does Center Club provide a place for 
members to socialize/interact and develop 
social networks and social skills? 

2. Does Center Club provide training and an 
encouraging environment for developing 
work goals? 

3. Do Center Club members find the club 
helpful in being able to remain out nf the 
hospital? 

4. In what ways and to what degree are 
members willing to get involved in the 
functioning of the club? 

"k What services could Center Club offer that 
would be helpful? 

6. What services do club members see the 
club presently offering? 

7. Does the club help members to use a widar 
range of community services? 

8. What is the perceived level of satisfaction 
that club members have with their Center 
Club membership? What are members 
dissatisfied with at the club? 

9. To what degree do members find the 
prevocational program favorable? 

10. What basic needs are preventing members 
from living satisfactorily in their 
community? 

11. To what degree does Center Club help 
members to fulfill their basic needs for 
living satisfactorily in their community? 

12. How much support from other mental 
health resources outside of Center Club 
are members getting? 

13. What are members' perceptions of th3 
changes at the Center Club, and in 
themselves since they've joined? 

14. How do club members feel about the pres- 
ent location of Center Clu! "'ersus a site 
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within a hospital or in a mental health 
setting? 

TECHNIQUES USED 

The questionnaire was developed based upon 
evaluation questions arrived at through meetings 
with staff and club members. At the first meeting 
staff were asked to list the questions they wanted 
to ask members, what they saw as Center Club 
goals, and what results Center Club should 
achieve. The survey was then discussed informally 
with a variety of club members, and a rough draft 
was written with the help of Dr. Spaniol. The 
draft was reviewed in a meeting involving both 
club and staff members and several new items 
were generated. The final survey was produced by 
the evaluator, using all of the information 
received at Center Club and incorporating one or 
two items from an earlier Center Club survey. 

The survey was distributed during a 10-day 
period in 1982. Both staff and club members 
distributed, monitored, and aided members in 
filling out the survey at the club. In addition, 
approximately 25 questionnaires were mailed to 
club members not presently attending the club 
who had previously been active, in order to get 
feedback from a wider range of participants. 
Slightly more than half of those mailed out were 
completed and returned. The final number of 
surveys analyzed was 99, or almost a 50-percent 
rate for completed returns. 

FINDINGS OF EVALUATION OR MONITORING 

Center Club members found the program quite 
valuable in helping them to remain out of the 
hospital, develop social networks and skills, and 
expand their use of available community 
resources. There was, however, more casual 
support for the success of the program in th3 
vocational area (i.e., providing effective prevo- 
cational training). The popularity of the club's 
social program in contrast to the prevocational 
program is not surprising considering the long 
history and predominance of the souial program. 
Another interesting finding was that almost 
one-third of the respondents indicated by their 
responses that they were nclear about how to 
take responsibility at the ciub. This is worthy of 
note since much of the emphasis in this type of 
program is on member responsibility. 

The vast majority of members indicated 
satisfaction with their club membership. The only 
area of dif satisfaction frequently mentioned was 
the lack of pay for their work. As mentioned, the 
club's policy on paid work had recently been 
changed and still appeared to be a sore issue for a 
significant number of members. Further, there 
were mixed feelings about the additional emphasis 
on the vocational area. Some members felt quite 
angry about the changes while others fCuAd the 
new programs quite beneficiah Significantly, 
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one-half of the respondents indicated that they 
would not come to the club if it were located in a 
hospital or mental health center setting. 

RECOMMENDATIONS 

The recommendations were addressed to the 
staff and active membership of the club. Specif- 
ically, the report suggested that some ways be 
found to address concerns in the prevocational 
area, either by calling special meetings, or 
through regularly scheduled groups. Through such 
discussions, these conce) is can be recognized as 
legitimate, possible alterations can b3 suggested, 
and implementation of needed changes can take 
place. The report also addressed the lack of 
clarity among some members regarding respon- 
sibility to the club, and suggested that outreach 
methods be developed to help these members to 
learn how to become involved, and how they 
might contribute. 

The remaining suggestions involved specific 
•^gramlng recommendations. In the area of 
-^cial/dating needs, more club dances, trips to 
other social settings, and use of specific soci 
skill groups were indicated as services that . .uld 
be considered helpful by many members. Recom- 
mendations for implementing a social skills group 
were suggested in the e valuator's report. 

STEPS TO ENSURE IMPLEMENTATION 

One of the goals in planning this evaluation was 
to try to include implementation steps as part of 
the overall process of the evaluation. This was 
done in two ways. First, the final item on the 
survey questionnaire asked respondents to 
indicate how they would like to see the 
information from the survey used. In this way, 
respondents themselves were involved in making 
suggestions about how the information gathered 
might contribute to progrann improvement. 
Secondly, during the planning phase the evaluator 
explained that a final report with recommenda- 
tions wculd be written and that some process for 
reviewing would be available. As a result, a 
committee made up of six club members and one 
staff member has been meeting weekly to review 
and discuss the recommendations and decide how 
they can most reasonably be implemented. The 
evaluator's report has suggested that the 
committee's recommendations take the form of 
specific, concrete plans and that the persons 
responsible for carrying thenn out be designated. 

EXTENT OF IMPLEMENTATION 

At this writing, all members of the evaluation 
review committee have read through the entire 
report and have met a total of six times. Follow- 
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ing each weekly meeting, the committee reports 
its progress at the Center Club council. The 
evaluation review committee plans to have its 
final suggestions for implementation completed 
by mid-September 1982. At that time, a formal 
presentation will be made to the annual general 
membership meeting held at Center Club. 

SPECIAL BARRIERS OR SUPPORTS 

The amount of time available to produce and 
follow through with the evaluation placed certain 
constraints upon the project. The actual survey 
period was reduced from 2 weeks to 10 days 
because of an administrative backlog. The survey 
instrument itself had a number of open ended 
questions which, though valuable, proved to be 
difficult for many respondents to answer. In 
addition, these questions were particularly 
difficult and time consuming to score because of 
the wide variety of responses. More multiple 
choice questions would have been preferable. On 
the positive side, both staff and club members 
gave generously of their time, especially in 
monitoring the administration of the survey and 
helping members who needed assistance (in 
reading, etc.). In addition, the staff was quite 
receptive to the overall process and club members 
responded with enthusiasm and a great deal of 
energy in filling out the survey. The fact that the 
outside evaluator was a former staff person 
probably contributed to the project's success. 
Staff and club members felt more comfortable 
and therefore less suspicious about the possibility 
of misuse of the information because they were 
assured that the evaluator was familiar with the 
program, its intent, and its members. 

RESOURCES AND COSTS 

The cost of this pro'^ct was minimal because of 
the voluntary services of the outside evaluator 
who contributed approximately 50 hours of work. 
This suggests the value of seeking out students in 
program evaluation to aid in planning and carrying 
out particular evaluation projects. Another 
possible resource for such projects is assistance 
from professional evaluators who may be willing 
to contribute some of their time, as Dr. Spaniel 
did, to serve as volunteer consultants. 

Additional costs of this evaluation included 
postage for the 25 surveys sent out by mail 
(including stamped return envelopes), and the cost 
of paper, mimeograph materials, and envelopes. 
Center Club staff provided their time to help in 
the organization and monitoring of the evaluation. 

ADDITIONAL COMMENTS 

No additional comments. 



46 

55 



13. Consumers Solicited to Review Children's Services 

Lois P. Burgner 

Florida Consortiian for Research and Evaluation 



SUMMARY 

A Citizen Review Group (CRG), comp-^sed of 
clients and agency consumers of a comn jnity 
mental health center (CMHC) participating in the 
Florida Consortium for Research and Evaluation, 
reviewed children's services. The CRG generated 
24 recommendations for submission to the 
center's governing board. Recommendations were 
both practical and feasible, and were, after some 
delay, acted upon by center staff. Recommenda- 
tions concerned quantity and quality of service as 
well as community relations. Although the unique 
circumstances of tPe center affected implemen- 
tation of the recommendations, 62 percent of the 
CRG suggestions were enacted upon within 8 
months and 72 percent within a year. The CRG 
demonstrated the benefits of citizen participation 
in the evaluation of CMHC services. 

TYPE OF ORGANIZATION 

Citizens from the service area who represented 
the interests of consumers and who had expertise 
in children's expertise, were recruited by the 
center to review evaluation materials generated 
by CMHC staff. The resulting group included 14 
persons: 7 staff members from schools and other 
agencies serving children, 6 parents of child 
clients, and 1 client. The CRG was chaired by two 
governing board members of the participating 
CMHC. The objective of the CRG was to make 
recommendations regarding children's services at 
the center. 

EVALUATORS OR MONITORS 

CRG agency members included school social 
workers, a child development center teacher, 
staff from Big Sisters and 3ig Brothers, and 
representatives from the State human services 
center. Family members of clients and one 
teenaged client participated. All members 
contributed to the discussions and most showed 
littie or no reticence to disclose their identities 
or experiences. The group varied in age, economic 
status, and occupation. 



For further information yvrite Charios Windle. Ph.D., 
Service System and Economics Research Branch. 
National Institute of Mental Health. Room 18-C-07, 
5600 Fishers Lane, RockvUle, MD 20857, (301) 443-4233. 



REASONS FOR THE EVALUATION 
OR MONITORING 

The CMHC, as part of a research project to 
study different ways for the community to 
participate in center evaluations (as then required 
by the Federal CMHC Act), agreed to participate 
in citizen review. The governing board, after 
reviewing the center's annual evaluation report, 
requested that a CRG address the evaluation 
reports of children's services. The consumer 
perspective was considered an important balance 
to the views of CMHC administrators regarding 
planning for children's services. 

LEVEL OF PARTICIPATION 

The consumer CRG was an ad hoc advisory body 
convened by the center to review and react to 
program materials and to '"oncentrate on client 
concerns. Recommendalions were generated and 
written by consumers. Center evaluation staff and 
governing board chairpersons acted as facilitators 
of discussions and as information resources. CRG 
members also added opinions to the recommenda- 
tions and acted as key informants for the impact 
assessment cond.'ctsd 8 months after the 
recommendations were submitted. 

TARGET OF EVALUATION 
OR MONITORING 

CRG members concentrated on children's 
services at the center which included: child and 
adolescent outpatient services, inpatient services, 
an educational center for day treatment, 
consultation and education services, and a 
program for pregnant adolescents. The center, 
located in central Florida, serves an urban/rural 
population of about 250,000. The orimary 
catchment area is a declining urban area, and 72 
pen nt of center clients report annual incomes of 
les --^han $5,200. Fees are based on ability to pay 
and no person is denied services. Consequently, 73 
percent of the center's funding comes from public 
revenues. 

PROBLEMS OR ISSUES EVALUATED 
O R MONITORED 

The center was concerned with soliciting 
consumer perspectives regarding the quality and 
appropriateness of center services for children. 
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Consumer perceptions of what was needed for 
children in the area would then be compared to 
State priorities for such services. 

TECHNIQUES USED 

Descriptions of the center and its services and 
children's programs, naps, and evaluation reports 
were provided to CRG members in an orientation 
packet mailed out before meetings began. CRG 
members participated in four weekly meetings 
and one followup meeting after 6 months. A list 
of State priorities for children's services was 
provided at the first meeting. Program overviews 
were presented by program directors and 
evaluation staff. Presentations by coui.cy 
recreation and transportation personnel and the 
Volunteer Action Agency director were requested 
by CRG members as pertinent to concerns arising 
from the discussions. The exchanges among 
clients, consumers of the agency's services, 
center staff, and governing board members 
produced a multifaceted list of concerns and 
recommendations. 

FINDINGS OF EVALUATION OR MONITORING 

CRG members were generally very pleased with 
the quality of the center's children's programs. 
Clients and family members praised staff and 
services, and reiterated the benefits of their own 
experience with center program?. CRG concerns 
centered on inadequate public funding of pro- 
grams for children and youth, particularly 
preventive services for adolescents. Consumers 
pointed out a greater need for collaborative 
efforts in behalf of residential drug treatment 
programs for adolescents, recreational opportu- 
nities, training for city recreation personnel, and 
transportation programs for youth. Further, the 
CRG was concerned that the community was 
often unaware of many of the programs of the 
centar. The consultation and education depart- 
ment was seen as an important link to the 
community for training and pubiio education 
about the center, mental health education, and 
outreach. 

RECOMMENDATIONS 

Recommendations presented to the governing 
board were primarily concerned with community 
relationships (79 percent) and the expansio''- of 
service*: co youth and parents (17 percent). The 
CRG ovfered practical and specific suggestions. 
For instance, CRG members drew from personal 
and professional experience to suggest names of 
persons to contact for possible publicity and 
contributions of space. Key locations for staff 
training and expanded services were listed. The 
CRG also pointed out the reluctance of residents 
to come to the mental health center for child- 
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care and parent-effectiveness classes, and 
suggested alternative sites, such as housing 
developments and churches. They encouraged 
collaboration between private and public organi- 
zations to promote recreational alternatives for 
adolescents. The prevailing limits of public 
funding were taken into account when the 
recommendations were written for submission to 
the center's governing board. Only four of the 24 
recommendations (17 percent) required additional 
money to implement; most could be implemented 
using only one to two staff members (79 percent); 
and a minimum of effort (63 percent). 

STEPS TO ENSURE IMPLEMENTATION 

The reason for making two governing board 
members the chairpersons of the CRG was to 
insure that CRG recommendations would have 
advocates before the board. Theoretically, if tne 
board members were committed to the CRG 
process, they could be expected to be enthusiastic 
about the recommendations, and to use their 
influence to enhance acceptance and imple- 
mentation of CRG suggestions. The letter of 
recommendations was presented to the governing 
board exclusively, with further dissemination 
subject to their review and approval. Delegation 
of responsibility to carry out the recommenda- 
tions was to be through the executive director of 
the CMHC to progran* staff. As it turned out, 
neither the chairpersons nor the center director 
lived up to these expectations and it was the 
intervention of a key person on the center staff 
that ensured implementation. 

A 6-month followup meeting provided CRG 
members with a report on the progress of their 
recommendations. After putting time and energy 
into evaluation review, citizens were entitled to 
know what actions resulted from their efforts. 
The 6-month meeting also served to reaffirm the 
center's commitment to respond to the recom- 
mendations. While these purposes were not 
optimally achieved in this particular CRG, there 
' ; ample evidence from the consortium experierice 
that followup .s a crucial component in assuring 
good implementation of recommendations. 

EXTENT OF IMPLEMENTATION 

Of the 24 recommendations, 17 were imple- 
mented fully or partially within 8 months. Four 
recommendations for outreach services were 
accept'^d but were not acted upon because of 
limited staff and time. Recommendations that 
v/ere inappropriate for the center to effect were 
tfv'anslated into more feasible actions. The 
implementation report to the CRG, described. ^ 
above, was well received. Interviews with 
participants revealed that the CRG members 
were pleased with the center's reception and the 
actions taken. The experience wa seen as a 
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positive coUabOiCtion and an effective means to 
promote communication between the center and 
its constituents. 

SPECIAL BARRIERS OR SUPPORTS 

The use of the CRG proved to be a viabJe 
mechanism for soliciting the community 
perspective and, in this case, the consume: 
perspective of center services. The structured, 
time-limited CRG meetings provided a relaxed, 
cooperative environment in which consumers, 
board members, and agency staff exchanged 
information, needs, and ideas for action. 
Involvement of governing board members in the 
CRG meetings lent credibility to the meetings 
and made the CRG rr embers feel that their voices 
would be heard. On the positive side, former 
clients were eager to take part in the discussion 
and offered several suggestions which later 
became recommendations. However, a decline in 
attendance among former clients at later 
meetings proved to be a disappointment. Speci- 
ficallyt the final meeting and the 6-month 
foilowup meeting were attended exclusively by 
agency personnel. The contributions of former 
clients during the first three meetings, however, 
was preserved and included in the letter of 
recommendations. 

The implementation of CRG recommendations 
was facilitated by the hiring of a key staff 
member within the 6-month foilowup period. The 
staff member, a public relations specialist, seized 
upon the recommendations and enthusiastically 
encouraged implementation. The staff member's 
dedicatio'i to the goals of consu.ner participation 
preserved the integrity of the process. Zhh 
commitment was even mor3 importar:t since the 
expected advocacy of the recommendations by 
governing board members did not occur, nor did 
the executive director cf the center assume 
responsibility for 3ini| that the recommenda- 



tions were carried out. The Iccter of recom- 
mendations was in limbo at the center until the 
public relations specialist initiated actions for 
implementation. 

RESOURCES AND COSTS 

Approximately 160 hours of staff time were 
spread out over 8 to 12 month?. A staff member 

•th either a BA or MA was required for assem- 
bling the evaluation materials, organizing the 
group, providing resource support, attending 
meetings, and so forth. 

ADDITIONAL COMMENTS 

The topics discussed by this CRG served to 
promote a spirited exchange of ideas by citizens 
concerned with their children's welfare. More 
mundane topics, although timely and perhaps 
crucial to center personnel, may require 
recruitment of community members with more 
specific expertise. The use of consumers, and in 
particular former clients, provided the center 
with information that could not be obtained 
anywhere else. The clients' perceptions and 
feelings about services and personnel provided 
important considerations for service planning. 
Although the former clients in this study appeared 
to enjoy the exchange of ideas with agency 
personnel, they might have felt more support 
from an all-client group and this, in tum, might 
have encouraged their attendance and 
participation. 

In { roups where anonymity is not an issue, 
ongoing publicity about the CRG's efforts and 
distribution of its recommendations to center 
programs, oversight organizaions, and the media 
can promote the public relations value of the 
process and the goals of the CRG. Implementation 
of the group's recommendations can be facilitated 
by those with the capability to take action or «"> 
encourage the center to act. The fruits of the 
CRG experience cannot ripen in the dark 



14. Statewide Comprehensive Mental Health Surveys 

Barbara Geddie 

Mental Health Association of North Carolina 



SUMMARY 

Monitoring and assessment of mental health 
services in both the community and instil i!*ions is 
a major role of the Mental Health Association of 



For further information write Barbara Geddie, Chair- 
person, Mental Health Association in Nor^h Carolina, 
5 West Hargett St., Suite 70S, Raleigh, NC 27601, (919) 
828-8145. 



North Carolina as part of its function as a citizen 
advocacy organization. In keeping with this 
mission, the professional advisory committee of 
the mental health association (MHA) worked with 
the North Carolina Division of Mental Health, 
Mental Retardation, and Substance Abuse to 
develop a plan for MHA participation in compre- 
hensive surveys of each community mental health 
program in North Carolina. In 1978, an MHA 
member participated in surveys in each of the 
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state's four mental health regions. This initial 
conclusion led to the agreement that an MHA 
member would be a full participant on each 
survey team and would be responsible for looking 
at community involvement and the responsiveness 
of mencal health centers to community needs. 
This is in contrast to team mem ers from the 
North Carolina Division of Mental Health whose 
focus is on meeting standards, record keeping, and 
the a'^ministrative and organizational almensions 
of the program, MHA members complete a 
checklist which organizes information from 
community referral sources and from the onsite 
visits. Areas out of compliance and recommenda- 
tions are included in MHA's official report which 
is attached to the division review. Local MHA 
chapters appoint or elect a survey team member. 

TYPE OF ORGANIZATION 

The Mental Health Organization of North 
Carolina is a volunteer service organization 
composed of 30 chapters throughout the State. 
MHA brings the perspectives of the client, the 
comn^unity, and the family to the survey process. 
The board of directors of the MHA is elected by 
local chapters. The purposes of cne association 
are to work for: improved care and treatment for 
mentally ill and emotionally handicapped persons; 
improved methods and services through research: 
prevention, detection, diagnosis and treatment of 
mental illness; and the promotion of mental 
health. 

EVALUATORS OR MONITOt^^S 

A survey guide developed and coordinated by 
the professional advisory board of the North 
Carolina MHA sets out requirements for MHA 
team members. Evaluators can be lay community 
members or professional members who are either 
retired or in private practice, but no survey team 
member can be an area board member because of 
conflict of interest. The MHA member on an area 
survey is an observer -participant in the area 
survey process. The MHA member is always part 
of r two-person suoteam and is not the leader of 
that subteam. The basic role of the MHA member 
is to assess community involvement in, and 
community attitudes toward, the area mental 
health program, specifically the use and role of 
volunteers, attitudes of other agencies (parti- 
cularly those that refer patients to the center), 
citizen awareness of the program, and mental 
health center responsiveness to community needs. 
Each MHA member is expected to comply with 
survey reportint:^ requirements, and to makes a 
report to the North Carolina MHA. 

REASONS FOR THE EVALUATION 
OR MONITORING 

Although the North Carolina Division of Mental 
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Health is responsible for reviewing mental health 
centers and agencies that receive Federal and 
State monies, the MHA believes that the citizen's 
perspective should also be represented. Such 
citizen advocates are necessary in order to 
identify areas of need that might not come to 
light during area surveys conducted by profes- 
sionals only. The MHA also believes that such 
advocacy might expedite the expansion of needed 
programs for which center staff might be hesitant 
to push. Also, the MHA member is concerned with 
determining, during a time of budget decreases, 
whether cuts are made according to the endoge- 
nous needs of the community or the popularity of 
a program with the staff. 

LEVEL OF PARTICIPATION 

Association survey members are full partners in 
every stage of the project from the planning 
session through the reporting session. They gather 
data from the communi' ' prior to the actual 
review and use such data as a foundation upon 
which to inquire about the provision of services. 
The association team member is not allowed to 
review the actual patient charts and see confi- 
dential data but can be involved in discussions of 
patient issues. 

TARGET OF EVALUATION OR MONITORING 

The surveys focus on the provision of service by 
community mental health centers. Currently, the 
State regional hospitals are not included in the 
surveys although this is a goal for the future. The 
surveys are designed to cover all program areas 
within the mental health centers and contact is 
also made by the association team member \Aath 
contracting agencies, referral sources, generic 
community agencies, and community leaders. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

While the State Division of Mental Health 
focuses on objective and measurable data to 
document delivery of services, the MHA member 
evaluates responsiveness of the program to the 
needs of the community. Consequently the 
opinions cind judgments of the volunteers are 
basically subjective although based on collected 
data. A key problem is recruiting citizens who 
have received service through the mental health 
centers. Tne State looks at contracts, records, 
and reports and talk3 with staff and affiliates, 
because of the issue of confidentiality, the MHA 
member is contrained ftom securing client 
records to evaluate service effectiveness. 

TECHNIQUES USED 

Telephone and face-to-face interviews with 
referring agents and contracting agencies provide 
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Information about accessibiUty, availability, and 
responsiveness of services. Community demo- 
graphic data is obtained to identify target groups, 
needs, and transportation routes to the'center. 
Interviews with samples of cliants are used to 
determine satisfaction with center services. 
Emergency services are evaluated by contacting 
the hospital, the outpatient department, a sample 
of private physicians, local counseling agencies, 
clergy, the police departnr 3nt, and tha magis- 
trate's office. 

FINDINGS JF EVALUATION OR MOMTORIMG 

Technically, most of the centers are in com- 
pliance in terms of documentation of se. .ce. Key 
issues that continue to be identified as problems 
include; (1) transportation to the center or from 
the center to regional hospitals; (2) program cuts 
and consolidation of services, which lead to 
difficulty in adequately meeting the needs of 
clients; (3) inadequate followup of patients dis- 
charged from the regional hospital to mental 
health centers or nursing homes; (4) need for im- 
proved public relations and community education 
regarding availability of services; (5) need for 
group homes and halfway houses; and (6) need for 
increased outreach to specific target groups. 
Further, the inadequacy of services for children, 
particularly inpatients, appears to be a problem 
statewide. 

RECOMMENDATIONS 

Based on the surveys. The North Carolina MHA 
Board is in the process of reviewing problem areas 
and making recommendations to the division of 
mental health as well as forwarding the informa- 
tion to local MHA chapters. Since this is the first 
year that the surveys have been conducted in a 
structured fashion, there is also a need for 
revision of forms and increased coordination 
betwc'^M the volunteers and the State divlc»on. 
Primarily, recommendations include: greater 
community education; increased services for 
children; improved coordination among regional 
hospitals, community centers, and nursing homes 
whbn patients are transferred from one facility to 
another; creative alternatives to meet the needs 
of the geriatric population confined to nursing 
homes and who are unable to get to the mental 
health center for service; improved followup; 
improved residential care in the community; and 
continued efforts toward smoother coordination 
among police and sheriff's departments, mental 
health centers, and regional hospitals. 

STEPS TO ENSURE IMPLEMENTATION 

The professional advisory board of the MHA 
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wlil review all reports from volunteers and write 
a detailed report to present to the MHA annual 
meeting. The chairman of the professional 
advisory committee and t'le president of the 
North Carolina MHA will meet with the director 
of the division of menial health to discuss the 
findings and work toward improved coordination 
of volunteers and survey teams. 

EXTENT OF IMPLEMENTATIO N 

The division of mental health has been recep- 
tive to MHA surveyors' recommendations. 
Further, the MHA will lobby for legislative 
changes as well as monitor to see if the results of 
the surveys are followed through. Already some 
revision of emergency services is occurring across 
the State as a result of the participation of survey 
team members from the association. 

SPECIAL BARRIERS OR SUPPORTS 

The survey team members need to be supplied 
with copies of the previous onsite review as well 
as an opportunity to meet with the director of the 
center to review standards in areas of com- 
pliance. Because the volunteers are usually 
employed, it is necessary that they have adequate 
notice of ii.3 meetings and be provided with 
orientation materials in order to prepare for the 
review and to make community contacts prior to 
the visit. The MHA, through its advisory commit- 
tee, is planning to provide greater in-depth 
orientation at the MHA annual meeting and has 
requested that the chapters identify their survey 
team members before that meeting in older for 
the team to receive appropriate orientation. 
Additional consumer involvement may be 
facilitated by contacting clients at local MHA 
social clubs, looking at notes in suggestion boxes, 
increasing media coverage, and publicizing a 
phone number to calL 

RESOURCES AND COSTS 

The volunteers supplied their own transporta- 
tion and meals; the mental health center did not 
have any expenses; and the State and local 
associations did not have any budgetary allow- 
ances for the surveys. 

ADDITIONAL COMMENTS 

It is important to remember that this struc- 
tured type of survey is in its first year of 
operation and that reviews in procedure and 
technique will be recommended. 
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15. Citizen Participation in Federai-State Site Visits 

Diane Rich 

Mental Health Association of Colorado 



The Mental Health Association of Colorado* in 
conjunction with the Alcohol, Drug Abuse, end 
Mental Health Administration (ADAMHA) of the 
Department of Health and Human Services 
(DHHS) and the Division of Mental Health of the 
State of Colorado, participated in a joint site visit 
project evaluating the 23 centers and clinics in 
Colorado that were receiving federal funds. The 
joint project began in 1976 and stopped tempo- 
rarily in 1981. The site visits resumed in 1982 
without representatives from DHHS. Thi*; case 
report describes the earlier stage of tno project in 
which Federal officials participated in tl-.e site 
visits. 

TYPE OF ORGANIZATION 

The Mrital Health Association of Colorado 
(MHAC) IS a voluntary nonprofit advocacy and 
education agency that works for improved care 
and treatment for the mentally ill, prevention of 
mental illness, and mental health education and 
re^'^arch. It is an affiliate of the National Mental 
Health Association. The site visit committee of 
the MHAC consists of six to eight citizen volun- 
teers, assisted by MHAC staff members. The 
citizens monitor the community mental health 
centers, clinics, and psychiatric hospitals 
throughout *:he State by participating in onsite 
evaluations. Committee members are appointed 
by the president of the board of directors of the 
MHAC. 

EVALUATORS OR MONITORS 

Over the years, members of the site visit 
committee have included mental health profes- 
sionals, a patient representative in a general 
hospital, a retired accountant, housewives, 
students, and association staff. All members 
received training in methods of evaluation and 
were observers on two site visits before becoming 
evaluators. Some of the evaluators have been 
Involved in the program since 1976. Each evalu- 
ator participates in a minimum of four visits per 
year. The citizen evaluators were accompanied by 
Federal and State officials on each site visit. 



For further information write Diane Rich, Mental 
Health Association of Colorado, 1117 Cherokee Street, 
3rd Floor, Denver, Colorado 80204, (303) 595-3500. 
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REASONS FOR THE EVALUATION 
OR MONITORING 

The Mental Health Association of Colorado, as 
an affiliate of the National Mental Heaith 
Association, is concerned wii:h improving the care 
and treatment of meritally ill persons and believes 
in the community mental health system. As the 
citizens* group representing consumers of services 
and those mentally ill persons who cannot speak 
for themselves, the association accepts the 
responsibility for assessing and evaluating 
community servicpi funded by the Government* 
The Federal and btate officers who also parti- 
cipated in the site visits were required by law to 
monitor the centers and clinics. 

LEVEL OF PARTICIPATION 

The site visit committee focused on twenty 
catchment area centers and three specialty 
clinics, a long-term treatment clinic, a children's 
clinic, and a clinic for Spanish-speaking (Hispanic) 
clients. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The site visitors monitored and evaluated 
multiple aspects of the center or clinic including: 
organization and administration, business 
management, facilities, quality assurance, 
emergency services, outpatient services, partial 
hospitalization, inpatient services, screening 
procedures, transitional halfway houses, foUowup, 
individualized treatment plans, range of treat- 
ment modalities, continuity of care, services for 
children and the elderly, community orientation, 
visibility, accessibility, preventive activities, 
consultation and education services, and coordi- 
nation with other agencies. The MHAC site 
evaluation committee participated in all aspects 
of the site visits but focused especially on the 
community aspects, visibility, accessibility, 
consultation and education, coordination with 
other agencies, services to children and the 
elderly, and the role of the board. Federal and 
State officials concentrated on other areas of 
program operation. 

TECHNIQUES USED 

The evaluation groups participated in develop- 
ing a site assessment instrurrent based on the 
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standards for the community mental health 
centers. The instrument is called the Review of 
the OnSite Evaluation Instrument (ROSEI). 
Centers were required to answer a preliminary 
questionnaire that was reviewed by th.*^ site 
assessors prior to the visit. A team of evaluators, 
composed mainly of Federal and State project 
officers, persons from other mental health 
centers, the Colorado Association of Community 
Mental Health Centers and Clinics, and the 
Mental Health Association then conducted the 
actual site visit. Onsite activities included 
interviews with staff, clients, and com 'nity 
agencies and viewing the various facilit and 
programs. In addition, the mental health profes- 
sionals on the team also reviewed records. 

FINDINGS OF EVALUATION 
OR MONITORING 

The site assessors wrote a report that pointed 
out the programs' strengths and weaknesses 
(concerns) and made suggestions for improve- 
ments. The report also contained a Federal site 
assessment rating form that rated the required 
areas as follows: (1) excellent, (2) meeting 
requirements, and (3) not meeting requirements. 
It included a summary checklist of required 
services, procedures and policies. The final report 
was written by the Federal or State officers and 
included observations from all the visitors. 

RECGNMENDATIONS 

Some recommendations were made to improve 
services, but centsrs were not penalized if they 
were not immediately and completely imple- 
mented. Other recommendations were made to 
bring the centers into cor.ipliance with standards 
and ensure continued funding. 

S TEPS TO ENSURE IMPLEMENTATION 

If a center was out of compliance, recommen- 
dations were made to bring it into compliance. 



and these steps were monitored by the Federal 
and State officials. If a center did not comply by 
a certain date, its Federal and State funding could 
be withheld. MHAC did not have any direct way 
of ensuring implementation on its own, but as part 
of the evaluadon team its recommendations were 
included in the site reports. 

EXTENT OF IMPLEMENTATION 

In geneial, all centers were in compliance or 
met standards satisfactorily after the visit. 

SPCCIAL BARRIERS OR SUPPORTS 

Initially, State and Federal project officers had 
to be convinced that MHAC participation on the 
site visits was appropriate. The MHAC volunteers 
soon proved their worth and in the last 2 years of 
the combined visits (after the State division of 
mental health dropped the comprehensive onsite 
visit55) the Federal project officers relied heavily 
on the expertise and commitment of the MHAC 
volunteers. 

RESOURCES AND COSTS 

Site volunteers donated between 100 and 130 
hours each year. MHA staff (clerical and profes- 
sional) also spent many hours. Two or three staff 
members went on two to three visits per year. 
The average yearly budget over the program 
period was $1,000. 

ADDITIONAL COMMENTS 

MHAC site visitors feel that the program has 
been ve*-y worthwhile. It has developed their skills 
and has given them a knowledge of the mental 
health system. The reports that have been 
generated helped MHAC in planning other 
advocacy projects and in lobbying the State 
legislature. It was considered a very good 
experience by both the MHAC and the Federal 
officers. 



16. County Board Assists in Assessing Local Services 

Jean Fiore 

Mental Health and Mental Retardation Advisory Board 
Northampton County, Pennsylvania 



SUMMARY 
The Mental Health and Mental Retardation 



For further information write Jean Fiore. R.N.. 
Executive Director. Visiting Nurse Association of 
Bethlehem, 520 East Broad Street. Bethlehem, PA 
18018, (215) 691-1100. 
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Advisory Board of Northampton County will be 
evaluating each of the eight services provided by 
the county mental health unit for the residents of 
the area. This case focuses on the first progranri 
evaluated, emergency services. The method 
chosen for the evaluation included interviews as 
well as a review of specific documents. Criteria 
were established that would assure the reviewers 
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that the service was effective, adequate, and 
Vunctioning eff iciently. The final report included 
strengths, weaknesses, and recommendations for 
improvenent of the service. 

TYPE OF ORGANIZATION 

The county board, made up of 13 persons, is 
advisory in nature unless otherwise specifically 
stated. Duties include review and evaluation of 
county mental health and mental retardation 
needs, service facilities and programs; assistance 
with the development of the annual plan; 
promotion of a better public understanding of the 
needs of the mentally disabled; and assistance in 
the development of policy. Members are nomi- 
nated by the county executive and appointed by a 
majority vote of the Northamptori County council. 
Nominees are community volunteers who have 
specific areas of expertise and are willing to 
commit their time. The term of the office is 
3 years, and membership is rotating. 

EVALUATORS OR MONITORS 

This evaluation was conducted by a comn 'ttee 
of six persons appointed from the advisory board. 
Committee members included social workers, 
clergy, nurses, a retired hospital administrator, 
and consumers. These areas of expertise were 
enhanced by other forms of community involve- 
ment such as school board membershrp, profes- 
sorship at an area college, and administration of 
other health agencies. The committee received 
technical asslstan^.e from Dr. Jonathan A. Morell 
through the K' rtnampton County department of 
human services. 

REASONS FOR THE EVALUATION 
OR MONITORING 

One of the duties of the advisory board is to 
review the programs and services Included In the 
county's annual plan. It was felt that more 
In-depth evaluations of programs should be 
completed than had been done In the past. It was 
determined that three services should be evalu- 
ated each year. A model was established to 
provide the reviewers with uniform tools, and 
criteria were developed for each service. After a 
service Is evaluated, the process can begin again 
using the same model. Therefore, a comparative 
analysis of evaluations over time will enable 
reviewers to document change and growth In each 
program based on recommendations from the 
previous evaluations. 



LEVEL OF PARTICIPATION 

The committee members were Involved In all 
stages of the evaluation. Initial meetings were 
held to discuss and determine criteria. The 
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method v/as developed Including questions to be 
asked during the Interviews. A session on role 
playing was held to give committee members 
necessary confidence In conducting Interviews. 
After each member completed his or her evalu- 
ation tasks, meetings were held to compile datu 
and to write the final report. County mental 
health and mental retardation staff were 
available as resource people in the development 
of criteria and to complete the necessary typing. 
Dr. Morell attended most sessions to provide 
technical assistance to the committee. 

TARGET OF EVALUATION 
OR MONITORING 

The first program the committee evaluated was 
emergency services, one of the eight services 
provided by the Northampton County mental 
health agency. This area was chosen because of 
Its community visibility and Its Interaction with 
other community resources. Emergency services 
are available 24 hourr a day for persons In need of 
Immediate psychiatric care. Services range from 
telephone counseling to hospitalization for 
observation, treatment, and close supervision The 
geographical area served Is Northampton County, 
which Includes both rural and urban areas. 
Emergency services staff receive calls from 
clients In distress, families, neighbors, police, 
physicians, and other human seivlce agencies. 
Telephone calls during the day are received at the 
units in Bethlehem or Easton. In the evenings, 
calls are received by an answering service and 
referred to the on-call person at home. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The evaluation of emergency services In the 
county focused on service delivery, program 
administration, staff supervision, staff scheduling, 
Inservlce training, and orientation. 

TECHNIQUES USED 

The evaluation tasks Included a review of the 
manual developed for emergency staff, review of 
administrative policy and statistical data, and 
Interviews with the administrator, supervisors, 
and service staff. Each committee member 
assumed a specific area to assess. For example, 
the person who reviewed the manual was search- 
ing for criteria that the committee determined 
were necessary for a functional staff manual. 
Another member reviewed statistical data and 
Interviewed the administrator. The remaining 
committee members Interviewed the supervisors 
and the staff. Each level of Interviews had 
specific questions based upon established criteria. 
The staff interviews were voluntary and confi- 
dential. The final phase of the project *vas to 
present the data according to the strengths and 
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weaknesses of the program, and to develop 
recommendations. 

FINDINGS OF EVALUATION OR MONITORING 

The committee members felt the entire process 
was beneficial not only because it resulted in a 
comprehensive evaluation and a model that can be 
used again, but the process was a good learning 
experience for the committee as well as staff. 

The program strengths identified were: (1) a 
cohesiveness exists among the staff at all levels, 
and the supervisory backup plan is adequate; 
(2) the on-call workers had a positive attitude, 
and all felt that the service was important; (5) the 
answering service is very functional; (4) the tools 
used for gathering statistical data are excellent 
for completing an analysis: (5) the staff manual is 
helpful; and (6) the paperwork associated with 
being oncali is reasonable. 

The areas of concern revolve around on-call 
duty at night, training and inservice education, 
and community resources. Nighttime on-call at 
the outset of the program was minimal; however, 
as the program grew, workers spent more hours 
handling calls, got less uninterrupted sleep, and 
burnout bee jt.id a serious threat. Further, the 
worker on call at night must also work s regular 
day shift, but the assignment is voluntary. Many 
staff members volunteer in order to earn more 
money, while others volunteer because they get 
job satisfaction from working on this type of 
service. 

The reasons for rejecting a nighttime on-call 
assignment were possible burnout, lack of 
confidence regarding nighttime on-call respon- 
sibilities, employee safety, and family respon- 
sibilities (primarily children). Safety and family 
are more likely to affect female employees. For 
example, husbands of female staff expressed 
concern for the safety of a spouse when she had 
to drive 50 or 60 miles during the middle of the 
night to a person in psychiatric crisis. Respon- * 
sibilities for children prevented some employees 
from taking nighttime on-call duty because the 
spouse was working night shift, and the children 
were young and could not be left alone. 

Transportation in emergency situations was also 
a problem. Each city, borough, and township has 
its own policies regarding whether ambulance 
squads and police will provide transport out of 
their jurisdiction. In addition, each hospital has its 
own admission system, and the on-call ^vorker 
must be familiar with all admission procedures as 
well as with the hospital staff on admission units. 
There is an unwillingness among area hospitals to 
accept mental health patients for emergency 
admission during the night. Of the four area 
hospitals used, one in particular is extremely 
difficult to work with and does not accept its 
equitable share of patients. Meetings on the 
administrative level have failed to solve the 
problems of this hospital. 
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RECOMMENDATIONS 

The committee suggested that staff orientation 
should be expanded to include outside workshops 
and seminars, as well as •*how-to** sessions. 
Further, short-term counseling on specific cases 
was recommended, preferably the day after the 
situation arose. Monthly case conferences were 
suggested to ensure service continuity since some 
clients call the emergency services more than 
once. These suggestions will enable more staff to 
take nighttime on-call duty thus alleviating 
burnout and insecurity on the part of the staff. As 
an alternative, the committee also suggested that 
it may be advantageous to hire additional staff 
specifically for this program. 

Hospital admission systems in all probability 
will not change. However, additional staff and 
improved training should enable staff to learn 
each hospital's system and develop a rapport with 
emergency room personnel. Transportation is an 
area that requires further work in order to find a 
solution to the problem. With respect to the 
hospital that does not admit sufficient clients, it 
is recommended that resolution be sought at the 
county administrative level. 

It is recognized by the committee that 
recommendations and areas of concern cannot be 
addressed and changes made without additional 
funding for the program. Presently, the emer- 
gency program has a low pric ^ty in the budget, 
and the committee recommenced that increased 
dollars be considered. 

STEPS TO ENSURE IMPLEMENTATION 

A detailed report of the evaluation was written 
and presented to the advisory board, chairperson 
of the county department of human services, the 
county executive and the state regional office. In 
addition, the committee met with county staff 
and discussed the report and recommendations. 
The report will also be included in the annual plan 
submitted to the state. 

Reevaluations over several years, using the 
same model, will enable the committee to 
complete a comparative analysis and to assess the 
extent of implementation of the committee's 
recommendations. 

EXTENT OF IMPLEMENTATION 

Implementation of recommendations has begun 
in the budget for FY 1982-83, and further changes 
will be made as the dollars are available. 
Recommendations made at the county executive 
level are not completed to date, however, a 
report will be requested at the next regularly 
scheduled meeting of the advisory committee. 

SPECIAL BARRIERS OR SUPPORTS 

The preparation of the criteria, model and tools 
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to be used took many hours of work on the part of 
th9 committee. Members spent several hours m 
formal meetings with Dr. Morell, technical 
advisor, as well as in the review of data and 
preparation for the meetings. However, this 
beginning will enable the committee to structure 
the remainder of the evaluations more easily. 

RESOURCES AND COSTS 

Formal committee meetings and work sessions 
amounted to 306 voluntary man-hours. Based on a 
75C-hour workday, this amounted to <i0.8 workdays 
or 8.16 workweaks. This figure is conservative in 
that the time spent by each member in reviewing 
data and preparinr for the sessions has not been 
calculated. The number of man-hours that the 
county staff, both professional and clerical, has 



spent on this project has not been maintained by 
the committee. 

ADDITIONAL COMMENTS 

The committee met with county staff to 
present and discuss the final report of the 
evaluation of emergency services. One of the 
other eight programs provided by the county is 
consultation and education services. This program 
provides inservice training on the organization of 
county services, including emergency services, to 
other community groups such as police, am- 
bulance corps, and other service providers. The 
county staff have recommended that advisory 
board members participate in the inservice 
programs provided for the community. 



17. A Consumer-Based Needs Assessment Examines 
Mental Health Services in One County 

Ellen Colom Deacon and Thomas M. Quilter 

Mental Health Association of Ohio 



S UMMARY 

The Mental Health Association of Ohio 
(MHAO), under a contract from the Office of 
Community Support Systems, Division of Mental 
Health, Ohio Department of Mental Health 
(ODMH), recruited a consumer group in Franklin 
County, (Columbus) Ohio, for the purpose of 
conductina a consumer-based needs assessment. 
Both MHAO and ODMH felt the need for greater 
consumer involvement in the design and planning 
of programs that ultimately affect persons served 
by Ohio's mental health delivery and volunteer 
advocacy systems. 

TYPE OF ORGANIZATION 

The Mental Health Associatiori of Ohio is a 
nonprofit charitable voluntary citizens' advocacy 
organization, whose mission since its founding in 
the ea^ly 19^0s has been to: (1) promote mental 
health, (2) prevent mental illness, and (3) improve 
treatment services for the mentally ill. MHAO, 
through its national affiliation and its 20 county- 
based chapters, works towa^^d the achievement of 
this mission through wide-ranging programs in 
public information and education, legislative 
action, research, and volunteer service. 



For further information -write Thomas M. O^ter, 
Executive Director, The Mental Health Association of 
Ohio, 50 West Broad Street, Suite 2440, Columbus, Ohio 
43215, (614) 221-5383. 



EVALUATORS OR MONITORS 

Two facilitators worked with the consumer 
group. They recruited members, conducted 
interviews, facilitated meetings, provided 
technical assistance, and compiled the final 
report. One facilitator was employed by the 
mental health association, the other by the Ohio 
Department of Mental Health. Neither were 
mental health clinicians. Another MHAO staff 
member observed the meetings to gain consumer 
suggestions for the design and development of a 
pilot MHAO advocacy program. Project PAVE 
(Patient Advocacy through Volunteer Effort). 

The consumer group was composed of nine 
residents of Franklin County. Criteria for referral 
and inclusion in the group included: 

• Ability to verbalize well 

• Leadership skills 

• Equal distribution of males, females and 
racial groups 

• History of hospitalization or some involve- 
ment with daycare programs within the past 
year 

Referral was made by aftercare coordinators 
from community mental health centers. Referral 
information included name, address, and tele- 
phone number. All additional information was 
provided directly by the individual. MHAO staff 



5.^ 

65 



conducted interviews with prospec ,ive members 
to define the program and to ascertain interest 
and willingness to serve. The group consisted of 5 
men and 4 women, with an age range of. 29 to 56 
years. Their educational levels ranged from high 
school to graduate school. Two members were 
involved in daycare programs, three were college 
students, one a full-time secretary, one a 
professional pianist, one a manager of a card and 
gift shop, and one was seeking volunteer work. 
One member had never been hospitalized; some of 
the others had been hospitalized 10 or more times. 

REASONS FOR THE EVALUATION 
OR MONITORING 

Recognition of the need for a consumer group 
grew out of discussions between MHAO and the 
Office of Community Support systems, ODMH. 
Both organizations felt the need for greater 
consumer involvement in the design and planning 
of programs that ultimately affect persons served 
by Ohio's mental health and volunteer advocacy 
systems. Missing from prior needs assessments 
was direct information from service recipients 
concerning the adequacy and availability of 
services upon discharge. MHAO, particularly in 
reference to Project PAVE, needed consumer 
assistance in shaping program development and in 
defining advocacy objectives and activities. 

LEVEL OF PARTICIPATION 

The consumer group met nine times to identify 
needs, assign priorities, suggest an action plan 
based on identified needs, and plan future 
activities of the group. 

TARGET OF EVALUATION OR MONITORING 

This study focused on consumer-determined 
needs in Franklin County. Franklin County is 
served by four comprehensive community mental 
health centers, one State psychiatric hospital, 
general hospitals with inpatient psychiatric units, 
private practitioners, and several private 
psychiatric hospitals. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The consumer group addressed a broad range of 
issues relating to the delivery of mental health 
services in Franklin County. Some of the major 
foci were public education, confidentiality, 
staffing, discharge procedures, service gaps, 
medication, and the acceptability of certain types 
of treatment. 

TECHNIQUES USED 

Several methods were used to identify needs. 
The Nominal Group Technique (NGT) was used by 
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the facilitators to assist with group problem- 
solving. The NGT encourages maximum partici- 
pation of all members in identifying priorities. 
The ••problem-centering- and -solution-finding** 
aspects of the project were stressed as tasks. The 
task, as identified for the consumer group, was to 
assess the county's needs and then to develop 
strategies to meet those needs. 

Individual interviews were also conducted using 
a checklist form with 22 subject areas. The 
purpose of the interviews was to conduct 
additional needs, to assess in more detail the 
problems encountered at the time of discharge 
and reentry to the community, and to determine 
what coping techniques, strategies and problem- 
solving actions were required to resolve these 
needs. Although community needs were discussed 
with the group as a whole, it was felt that more 
specific information could be generated through 
individual interviews. 

FINDINGS OF EVALUATION 
OR MONITORING 

The NGT process proved to be a fruitful method 
for identifying the major unmet needs of the 
mental health consumers. Attendance at each 
meeting was remarkably good, communication 
levels were consistently high, and motivation to 
participate was maintained. 

Lists of needs, ranked by priority, y^eve 
generated by the consumer group. In terms of 
importance, consumers placed top priority on the 
urgent need for public educatior about mental 
health so that the stigma of mental illness could 
be reduced. The consumer group felt strongly that 
public attitudes regarding mental illness must be 
changed. Indeed, they placed this need ahead of 
more basic and individual needs such as jobs or 
housing. Improving the quality of professionals in 
the hospital was also seen as a pressing need, as 
were improvements in the quality of the hospital 
environment and treatment approaches. Specific 
issues included over-medication, therapeutic vs. 
punitive approaches, lack of individualized 
treatment, high-carbohydrate food, and the need 
for privacy such as showers in each hospital room. 
In total, the identified hospital needs concerned 
changes in the institutional ethos of State 
psychiatric facilities. 

On the community side, consumers indicate 
that in addition to public education to reduce 
stigma, they also need help getting jobs, sup- 
portive friendship at critical times, and continuity 
cf care. Other critical needs (in descending order 
of priority) included: help in getting along in the 
social world; temporary jobs before assuming 
full-time positions; legal safeguards on privacy; 
recognition as a person with worth and dignity; 
and greater responsiveness from the Social 
Security Administration. 

It is important to note that the consumers in 
this group were selected to represent the 
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concerns of consumers in Pranklin County only* 
However, it may be that their concerns may be 
generalized to a iarger population of consumers. 
More groups such as this one are needed so that 
comparative analyses aie possible. 

RECOMMENDATIONS 

The consumer grcup developed a number of 
recommendations to deal with ne'^ds identified 
within the Cojnty. The recommendations included: 
working with the news media to impro /e the 
image of former patients; developmef)t of 
employment-related programs; a self-help 
hotline; a buddy-system telephone network; 
improved procedures for confidentiality'; 
increased dialog between consumers anu hospital 
administrators; and placing consumers on hospital 
advisory boards. 

STEPS TO ENSURE IMPLEMENTATION 

Project staff members wrote a detailed report, 
subsequently approved by the consumer group, 
that included descriptions of the needs assessment 
process, techniques, results, and recommi'^nda- 
tlons. This report was submitted by MHAG to the 
Division of Mental Health, Office for Community 
Support, Systems Planning and Development. 

EXTENT OF IMPLEMENTATION 

MHAO Implemented the project recommenda- 
tions in the following ways: 

• Findings and recommendations were used in 
the design and implementation of Project 
PAVE, now underway in Franklin County. 

• Findings and recommendations have been 
used to help formulate the MHAO*s overall 
legislative program and advocacy thrust. 

• Findings and recommendations have been 
used extensively in the MHAO's internal and 
external training program as well as in public 
education programs. 

• MHAO has responded to nationwide requests 
for copies of the project results. Hopefully, 
this will stimulate the development of similar 
needs assessment processes in other areas. 

SPECIAL BARRIERS OR SUPPORTS 

Although the initial purpose of the consumer 
group was task oriented, the focus of the group 
appeared at times to be highly therapeutic. It is 
not paradoxical to find a work-oriented group 
producing such benefits; but it is questionable as 
to how much staff support and involvement with 
group members is desirable before the group 
begins to reflect staff opinions. 
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By the end of the formal consumer group 
meetings, the project task has been superseded by 
a strong group alliance providing peer support. 
Although future plans included the development 
of a constitution and bylaws for the group to 
continue as a consumer coalition, the group failed 
to remain cohesive after the formal activities 
ended. This failure can be attributed in part to 
the group's distrust of organizational help, 
including that from MHAO. 

RESOURCES AND COSTS 

MHAO expended $9,000 in direct costs to 
conduct the consumer needs assessment. The bulk 
of this was for project staff over the 6-month 
period. Other direct costs included supplies and 
transportation for consumers. Indirect costs for 
other MHAO professional staff time and other 
items were not calculated. In addition, indirect 
costs for professional staff time from ODMH 
were not calculated. 

ADDITIONAL COMMENTS 

MHAO is impressed with the initial findings of 
the needs assessment project. These findings 
reinforce the concepts developed by citizen 
advocates through the mental health association. 
The concerns expressed by the consumers 
regarding the hospital environment and the 
adequacy and availability of community-based 
services have been a primary concern of the 
association. The importance of the work done by 
consumers is evident and demands the attention 
of all people concerned with i'nproving the mental 
health system and the attitudes in our society 
that serve to perpetuate the negative aspects of 
that system. 

Additionally, the project identified the issues of 
community attitudes and stigma that have been a 
program priority of MHAO for considerable time. 
In light of these findings, MHAO believes that 
much can be accomplished through the continued 
cooperation of consumers. 

MHAO strongly recommends that individual 
consumers willing to serve at both the chapter 
and State levels should be considered for appoint- 
ments on mental health association boards and 
committees. MHAO will continue co utilize 
consumer reactions as a necessary component in 
both the development of Project PAVE and in 
other statewide programing. 

The MHAO supports: 

• involvement of consumers at all levels of the 
association and provides for this by appoint- 
ing consumers to the board of trustees and to 
committees. 

• The formation of consumer group*? both for 
self-help and for organizing to work for 
system changes. 
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18. Community Group Analyzes Mental Health Center Services 

Rita T. Parle 



Mid-Nebra^a Community Mental Health Center 



SUMMARY 

A group of community leaders sponsored by the 
Chamber of Commerce* county medical society 
and board of trustees of the community mental 
health center established a study committee to 
research the functioning of the center. A 6-month 
study produced a report with U recommendations 
for the center board* U for the executive 
director of the center* and 6 for the county board 
of supervisors. The board was advised to limit its 
activities to policymaking* establishing annual 
audit procedures* and developing long-range 
planning procedures. The center director was 
advised to tighten internal management controls* 
and the county supervisors were advised to 
develop policies to eliminate conflicts of interest. 

TYPC OF ORGANIZATION 

The Mid-Nebraska Mental Health Center 
(MNCMHC) is operated by a volunteer board of 
trustees representing the 12 counties in the region 
served. Trustees are selected hy their county 
boards of supervisors or comrassioners for 3-year 
terms. The trustees elect their own officers. All 
major staffing and policy decisions are reviewed 
and acted upon by the trustees. Any citizen 
interested in the center and its operations is 
encouraged to contact the trustee from his or her 
county. An advisory committee assists the 
trustees In matter? of programming and planning 
to meet the needs of thv^ central Nebraska region. 

EVALUATORS OR MONITORS 

In order to explore) issues that had grown up 
around the operations of the center* the board of 
trustees appointed a study committee. The 
chairman of the committee was a local banker. 
Other appointees included: an attorney* the vice 
president and general manager of a local radio 
station* a local physician* a retired colonel* the 
director of the local Chambe; of Commerce* an 
elementary school principal* a current member of 
the board of trustees of the center* a certified 
public accountant* a local business executive* the 
assistant principal of a senior high school* and the 
newly hired executive director of the center. 



For further injormatian write Rita T. Parle, Executive 
Director, Mid>Nebraska Community Mental Health 
Center, P.O. Box 1763, Grand Island, NB 68802, (308) 
381-5250. 



RE ASONS FOR THE EVALUATION 
O R MONITORING 

On April 21, 1980, at their regular meeting, the 
center board of trustees responded affirmatively 
to the suggestion of the chairman to appoint a 
study committee to investigate controversies that 
had surrounded operations of the center for 
several years. This action followed months of 
considerable debate .^garding budgets and 
personnel at the center. These discussions 
occurred in many settings such as meetings of the 
center board of trustees, meetings of county 
boards and other civic groups throughout the 
community. 

LEVEL OF PARTICIPATION 

The members of the study committee con- 
ducted an intensive independent study, meeting 
every week for about 4 hours in full session, and 
privately in subcommittees. Center staff as Isted 
in providing research and duplication of relevant 
materials. The Chamber of Commerce provided 
secretarial assistance. It was understood that 
there were to be no restrictions that would limit 
the inquiry of the committee. 

TARGET OF EVALUATION OR MONITORING 

The subject of the assessment is a comprehen- 
sive community mental health center located in 
Grand Island, Nebraska. The center provides 
services to 12 counties covering 8,200 square 
miles and a population of 1 12,000. Three satellite 
offices assist the main center in outreach to the 
rural areas. 

'P ROBLEMS OR ISSUES EVALUATED 
.R MONITORED 

The objectives of the study were as follows: 

• To review all relevant documents governing 
the formation and establishment of the 
center and its board of trustees, including 
items such as any current arrangements, 
composition of the board of trustees and 
characteristics of the counties involved, 
procedural rules of the board, and Federal 
and State laws and regulations 

• To review the mission of MNCMHC including 
the twelve primary services provided 
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• To review center planning activities, 
including the 5-year plan 

• To review the organization and staffing of 
the center 

• To review budget and finances 

• To review the level of services to clients, 
including the magnitude of current and future 
caseloads 

• To review the complaints and criticisms of 
staff, professionals, the general public, 
clients, and others 

• To review physical facilities available to 
MNHMHC 

TECHNIQUES USED 

The above tasks were assigned to subcommittee 
teams made up of study committee members. 
With the assistance of center staff, a great many 
documents which bear upon the scope of the study 
were prepared, duplicated, and distributed to each 
of the study committee members. Each subcom- 
mittee was directed to review the documents 
provided in addition to attending all of the 
meetings of the committee and hearing all of the 
testimony to be offered. In order to solicit 
testimony from the public, the corrr-'ttee 
prepared and mailed a release to mu ;ia within the 
12-county area. The release was used by the 
Grand Island DaUy Independent and local radio 
stations. Testimony was confidential and those 
wishing to appear were given appointments. The 
process required a substantial number of meet- 
ings. Twenty-three people appeared to testify in 
person and one individual gave testimony by tape 
recording. Several letters were also received by 
the committee which constituted comment or 
testimony. The statements received were from a 
wide variety of individuals, including professionals 
in the community who have reason to use the 
services of MNCMHC, lay members of the public 
who have used center services, current center 
staff members, and former center staff members. 
In addition, testimony was received from 
representatives of Government, and a former 
member of the board of trustees. 

FINDINGS OF EVALUATION OR MONITORING 

The findings of the study committee can be 
summarized as follows: 

• The staff is extremely dedicated and 
concerned about delivering quality com- 
munity-based mental health care to the 
residents of the 12-county area. However, at 
times, this dedication was overshadowed by 
the fact that persons were beinn .jacked to 



ERLC 



perform tasks for which ihe, oid not have the 
educational background. In other words, the 
overall staff quality, in terms of educational 
proficiency, did not keep pace over the years. 
In fact, the management quality of MNCMHC 
during the past administration deteriorated, 
and the quality of the staff deteriorated 
along with it. 

• The board of trustees is generally dominated 
by one or two members appointed by the 
local county board of supervisors. This 
dominance coupled with the long-standing 
conflict between the local board and the 
center has served to create an unfavorable 
political atmosphere that reduces the ability 
of the staff to deliver high-level care to the 
patients of the center. In fact, the political 
nature of the board of trustees and its 
influence on the center have caused job 
insecurity for current and former staff. 

• There was a substantial number of conflicts 
of interests among staff, trustees, and other 
board members stemming from relationships 
with a variety of human services programs 
related to the center. 

• On the positive side, the board of trustees 
appointed in April a new executive director 
for the center. Staff morale seems to have 
improved to some extent. In addition, certain 
other staff requirements were filled including 
the hiring of competent and well-trained 
therapists. 

• The coordination of aicohol treatment 
services in the 12-county area is a problem. 
However, the efforts of the board of trustees 
to define the level of alcohol services to be 
performed by the center and also to encour- 
age cooperation between center start ana 
other regional alcohol programs should be 
commended. 

• A number of other positive steps were being 
taken to improve the physical facilities of 
the center and to improve public relations in 
the comi.. unity, and with other medical and 
professional health care providers. 

All things considered, the study committee 
resolved that services pros-rded by MNCMHC are 
vital to the community ana shoUid be continued in 
the future. They saw the center's potential for 
becoming a model institution serving the public. 

RECOMMENDATIbiNlS 

The study committee presented the board of 
trustees with 14 recommendations for themselves, 
14 for the new director, and 6 for the county 
board of supervisors. In summary, the board of 
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tmstees was advised to delegate to the new 
diructor full authority and responsibility for 
management and participation in policymaking. 
The board was also advised to establish an -annual 
operational audit and cost study, a long-range 
planning process, and to seek more cooperation 
and consolidation with other human service 
agencies* The new director was to develop a 
strong organizational structure with internal 
controls, and to tighten and strengthen casework 
evaluation and supervision. Finally, the county 
board was to prevent actual or implied nepotism 
and conflcts of interest, and to resolve the 
long-standing problem of leasing the center. 

STEPS TO ENSURE IMPLEMENTATION 

The report was printed, presented at a public 
meeting of the board of trustees, and released to 
the local media. Members of the study committee 
attended subsequent board meetings to be sure 
their report was addressed. The Chamber of 
Commerce continues to monitor the progress of 
the centor through one member who became a 
member of the board of trustees. Several other 
members of the study committee have been 
appointed to the center's advisory board and thus 
continue to be apprised of progress. 

EXTENT OF IMPLEMENTATION 

Although several recommendations will take 



years to implement, the study committee agrees 
that most of the deficiencies have been cor- 
rected. In particular, the issues of conflict of 
mterest and internal dissension have been 
remedied. The board of trustees and the executive 
director have a close and cooperative relationship 
which has assisted in the entire process. The 
hiring of new ccaff members has also had a major 
impact on quality and efficiency of services. 

SPECIAL BARRIERS OR SUPPORTS 

About a month after the study began, the city 
was struck by seven tornadoes which made Grand 
Island a disaster area. Although the board of 
trustees allowed another 6 weeks for the study, 
time constraints due to disaster work precluded a 
very thorough study. On the positive side, the 
evaluation was supported by intense interest from 
those working on it and by an unusually competent 
and responsible group of civic leaders. 

RESOURCES AND COSTS 

The Chamber of Commerce and medical society 
paid for any expenses other than duplication of 
center materials. 

ADDITIONAL COMMENTS 

If this study is done again, a year's time would 
be better than 3 or 4 months. 



19. Citizens Monitor Services for Nursing Home Residents 

Christine Anderson, Michele Bolienbeck, and Iris Freeman 

Nursing Home Residents* Advocates, Citizen Advocacy Project 
Minneapolis, MN 



SUMMARY 

The citizen advocacy concept was originally 
designed in the mid-1960s by Dr. Wolf Wolfens-- 
berger as a means of advocating for developmen- 
tally disabled citizens (Wolfensberger and Zauha 
1973). Citizen advocacy is the pairing of a trained 
volunteer with a person in need of both compan- 
ionship and protection. The model has been found 
very appropriate for meeting the individual needs 
of nursing home residents. 

Citizen advocates serve in a variety of roles, 
both formal and informal. Formal roles include 
such activities as representation to obtain public 
benefits and protective or professional services. 



For further information write Michele BoUenback, 
Route 3, Box 328, Brainerd, MN 5640U (218) 829-8298; 
or Wolf Wolfensberger, Training Institute, 805 South 
Grouse Ave.. Syracuse, NY 13210, (315) 423-4264. 



Informal roles include concern, friendship, 
guidance, and affection. Advocates may assist the 
nursing home resident in obtaining services; but, 
the essence of citizen aavocacy is a one-to-one 
voluntary relationship. A citizen advocate may be 
the resident's only friend. 

Of equal significance is the advocate's role as 
service monitor for the community because his or 
har repeated presence delivers the message that 
outsiders are mindful of residents' safety and 
rights. The advocate also brings an often-missing 
message back to the community, namely the daily 
experience of the frail older person living in 
poverty. The hope of the program is that ex- 
panded public awareness can grow into public 
accountability. The potential for change, then, 
rests not only in individual relationships and 
actions, but in the cumulative effect of advocacy 
involvement over time. Throughout sone of the 
following sections, one case example is used to 
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illustrate some of the characteristics of citizen 
advocacy. 

TYPE OF ORGANIZATION 

Nursing Home Residents* Advocates is a 
consumer service agency devoted solely to 
problems in long-term care. The organization 
provides services to current and potential nursing 
home residents and their families and friends 
through advocacy casework, community action, 
information, and support. The agency is staffed 
with soecialists in social work, long-term care 
regulation, alternatives to institutional care, 
financial programs, research, community 
outreach, and residents' legal rights. 

EV ALU A TORS OR MONITORS 

Citizen advocates are all t; Des of people. No 
professional criteria are used in screening. 
Volunteers are drawn from all segments of the 
community. The volunteers eve interviewed to 
determi. e their commitment to the program's 
mission and to assess their basic comm*jnication 
skills. If accepted, their next step is an 8-hour 
preservice orientation in long-term care, 
residents* rights, and complaint resolution, the 
basics for a lay person's involvement in a complex 
ar.cJ often confusing health care oystem. Inservice 
help with individual problems is available later. 

REASONS FOR THE EVALUATION 
OR MONITORING 

One case example explains why citizen 
advocacy is important. A referral of a nursing 
home resident was made by the organization's 
casework staff. The caseworker expressed 
concern that the person, who had recently been 
placed in a nursinq home, had little cr nc 
regularly visting family. Coupled with lack of any 
short-term -^mory and progressive vision loss, 
her vulnerability would be very high in th3 best of 
circumstances. Given the public record of the 
facility where she was living, the neeo for 
immediate service and monitoring of care was 
apparent. The citizen advocacy coordinator's 
initial visit with this resident revealed cuts and 
bruises above her right eye and bruises and 
swelling around her upper lip. This finding, 
combined with the above-mentioned factors, 
made her a top priority for matching with a 
concerned jnd informed volunteer. 

LEVEL OF PARTICIPATION 

Each volunteer's monitoring/evaluation 
activities are tailored tc the needs of the 
individual resident. The volunteers are provided 
with any information they feel is i*elevant (and 
which is either public information or released by 
the resident). They receive support and informa- 



tion from the program coordinator as well as 
others on the professional staff. 

TARGET OF EVALUATION OR MONITORING 

This type of monitoring/evaluation is don." in 
nursing homes throughout the Twin Cities 
metropolitan area. The homes are choser; not so 
much for the risk factor or reputation but through 
prest*- criteria. The critical factor is that a 
vulnerable resident who requests service lives 
there. 

In the case of the particular resident described 
above, the facility had a record of being deficient 
in compliance with several State licensure and 
Federal certification standards. Though at ono 
time it had offered high-quality services, the 
facility began to deteriorate in 1977 when 
ownersnip changed. Since that time, its record has 
been consistently marginal. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The primary problem addressed b/ the citizens 
advocacy program Is to ensure that nui'^ing honr.p 
residents receive an acceptable level of ^are 
based on their entitlements in law and regulation. 

TECHNIQUES USED 

During tt ^ first few months of the advocate's 
relationship ^ith the resident described above, 
the resident was ambulatorv, although she needed 
assistance. Even though sht lacked short-term 
memory (she often confused the advocate with 
her daughter), the resident was able to converse 
with the advocate and at times appeared to 
remember who the advocate was. Some of the 
things which the resident and the advocate snared 
wtfie: 

• Sitting together talking 

• Going for walks 

• The company of the advocate's 2-year-old 
daughter 

• Holding hands and touching 

• Looking at magazines 

During the next few months a series of events 
caused the resident's condition to become more 
serious. She suffered depression after the death of 
her only child. She became weaker physically. She 
seemed to lose some of her hearing. She fell out 
of bed. She developed fluid on her knee and was 
sent to the hospital for tests and observation. 
When she returned from the hospital, she seemed 
more withdrawn and did not speak to the advo- 
cate. Today, their activities are: 
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^ The advocate talking to her 



SPECIAL B/, TRIERS OR SUPPORTS 



• The advocate taking her for rides in a 
wheelchair 

• The advocate sendiny her flowers, cards, and 
candy 

FINDINGS OF EVALUATION OR MONI'i JRING 

Despite her condition, the resident received no 
occupational therapy . It was questionable whether 
she had any therapy. After her hospitalization the 
resident moved to a more heavily supervised 
floor. None of her personal effects were visible. 
Either they had not been put into her new room or 
they were in the closet. The environment was new 
and impersonal, the i/ery opposite of appropriate 
care for someone who is already disoriented. The 
resident did not have adequate personal clothing 
and did nui seem aware that she had a small 
personal allowance to buy new clothing. Personal 
clothing appeared to be lost, or handled very 
carelessly. 

RECOMMENDATIONS 

One way to begin to protect this resident's 
rights to health services (therapy) and other rights 
(personal clothing, personal funds) is having the 
advocate represent the resident in negotiation 
with facility staff and public agencies. 

STEPS TO ENSURE IMPLEMENTATION 

Being developed. 

EXTENT OF IMPLEMENTATION 

As yot urtKnOwii. 



The things that have made remedies in this case 
difficult are: 

• The resident's physical and mental limitations 

• Staff turnover which makes it practically 
impossible to establish any continuity in 
problemsolving 

RESOURCES AND COSTS 

Staff: 1 full-time project coordinator, $12,500 
annually; 
Volunteer hours: ^0 hours; 
Length of involvement in this case: 13 months. 

ADDITIONAL COMMENTS 

There are several things that the advocate feels 
she could have done differently: 

• Spend more time with the resident 

• Visit once a week 

• Talk more 'o the social service agency 

• Try to get in contact with her family (if any 
exists) 

• Be a stronger advocate 

• Not assume staff is uncooperative 

The advocate feels that residents should be 
visited regularly (once a week) on the same day 
and time. 



20. Council for Community Services Agency Evaluation 

Beverly Kreis, Dave Heden, Steve King, Joan Merdinger, and Mary Kolsky 

Council for Community Services, Providence, Rhode Island 



SUMMARY 

The Council for Community Services (CCS) 
evaluation task force evaluates community 
delinquency prevention and treatment programs 
funded by the Rhode Island Governor's Justice 



For further information -write Beverly Kreis. Chief, 
Department of Children and Their Families. 610 Mt. 
Pleasant Avenue. Providence. RI 02908. (401) 277-6525. 
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Commission (G JC). One example of such a 
program is the Sophia Little Independent Living 
Program, evaluated by a task force in 1982. Three 
task force members and the CCS evaluator served 
as a miniteam to assess the program's effective- 
ness as based upon client success rates and staff 
judgments about client outcomes. A review of 
client characteristics, program services delivered, 
and overall operation of the program was also 
conducted, together with a comparison study of 
other similar programs and their outcomes. 
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TYPE OF ORGANIZATION 

'he Council for Community Services is an 
independent planning, evaluation ard research 
agency, working under local, State and Federal 
contracts in the human services ^ield. The CCS 
Evaluatioii task force was organized by CCS in 
March 1974, and is composed of approximately 20 
human service professionals in such fields as 
mental health, family services, juvenile j'jstice, 
law enforcement, education, and community 
organization/planning. The task force is chaireo 
by a family court judge and there are also several 
lay repre^^entatives. The task force has a dual 
role: (1) to actively participate on miniteams that 
perform evaluation activities, and (2) to serve in 
an advisory capacity in overseeing the work of 
miniteams and in reviewing and approving 
evaluation reports. 

EVALUATOr^S OR MONITORS 

The evaluation of the independent living 
program was conducted by the CCS evaluator and 
three volunteer task force members. Two of these 
members work for the Rhode Island family court 
as intake counselors and one member works for 
the Providence school department as an admin- 
istrator. The work of the miniteam was reviewed 
and approved by the full task force. 

REASONS FOR THE EVALUATION 
OR MONITORING 

This evaluation was requested by the Rhod** 
Island Governor's Justice Commission, (GJC) as 
part of an ongoing contract with the CCS 
evaluation task force to evaluate rour or five of 
the GJC-funded juvenile justice projects per year. 
The independent iiving proaram's funding was due 
to expire at the end of the year and an evaluation 
was seen by both the GJC and the program as 
desirable, whether for seeking new monies or 
making the decision to discontinue the program 
when GJC monies are terminated. 

LEVEL OF PARTICIPATION 

Task force members serving on evaluation 
miniteams participate actively in site visits to the 
various programs being evaluated. Specific 
activities in which they are directly engaged are 
record review, data gathering, staff and client 
interviews, and, to a lesser extent, provision of 
technical assistance to program staff. Depending 
upon the ^.^pe of program being evaluated and the 
level of evaluation required, miniteam members 
also assist in pre-post sr jdies. instrument design, 
und the interpretation of test results. The staff 
evaluator generally plays the major role in: 
formulating the evaluation design (with the aid of 
miniteam members); analyzing program data; 
writing site visit reports and the final evaluation 

64 



report; formulating recommendations for program 
improvement and internal evaluation systems; and 
extending iechnical assistance both durin^^ - id 
after the evaluation process to aid in the 
implementation of recommendations. Task force 
members serving on miniteams have continual 
opportunities for influencing these latter 
activities. Depending upon members' particular 
areas of expertise, their impacts can be both 
influential and significant. The full task force 
provides ongoing revi»?w and commentary for the 
miniteams and staff evaluator. 

TARGET OF EVALUATION OK MONITORING 

The Sophia Little Home for Girls in Cranston, 
Rhode Island, operates the independent living 
pronram for girls, ages 16-19, who have comple- 
ted Che residential (group home) phase of the 
program and are capable of living on their own 
and holding a job. The girls in the program have a 
history of family .ind emotional problems and 
were in foster care placements or State 1 5oi- 
dential facilities prior to entering the residential 
component. The Sophia Little program has been 
operating for over 75 years, first as a home for 
unwed mothers and later as a 20-bed group home 
for girls referred by the State department for 
children and their families (DCF) or the juvenile 
justice system. The program is privately run and 
receives support from the United Way, private 
donations, and contracts with DCF and GJC. The 
GJC-funded independent living program was 
initiated in January 1980, as a statewide program 
tc serve approximately eight girls per year. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

ihe evaluation focused on the outcome of girls 
served in the independent living program. Four 
impact variables were selected by program staff 
and the C2S evaluation team as indicators of 
success. These variables were: (1) self-esteem, (2) 
decisionmaking skills/responsibility, (3) self- 
control/self -discipline, and (4) interpersonal and 
family relationships. Objective tests (befure, 
during and after) and subjective assessment (e.g., 
client and staff interviews, and observation) were 
used to measure client success. Additionally, the 
miniteam reviewed program operations, manage- 
ment, and recordkeeping procedures relative to 
the quality of client services provided. 

TECHNIQUES USED 

The CCS evaluator and miniteam members 
made regular site visits to the program to develop 
the evcluation design, agree on appropriate 
evaluation instruments and procedures, review 
client records, assemble client profiles (charac- 
teristics) for each girl in the study, and conduct 
staff and client interviews. Evaluation activities 
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were conducted by program staff, the CCS 
evaluator, aid the cit'»zen evaluation miniteam. 
Each client was testeo at three points over a 
6-month time period and test results were 
analyzed by the CCS evaluator and miniteam. 
Program staff filled out behavior checklist forms 
on each client at the same three time intei yals as 
the client tests, and their assessments were 
compared with client test results. A client 
interview form was developed by the CCS 
evaluator and miniteam members and uiterviews 
were conducted with each client at the conclusion 
of the evaluation period. Results of these 
interviews were compared with client tests and 
staff assessment results. Overall program 
effectiveness was analyzed on the basis cf the 
above measures, and a final report was prepared 
by the CCS evaluator. Joint recommendations 
were formulated by the CCS evaluator and the 
miniteam members, and the complete evaluation 
report was reviewed and approved by the entire 
CCS evaluation task force. Suggestions from the 
CCS evaluation task force were included 
throughout the evaluation process and they had 
the opportunity to review staff and miniteam 
activities at regular monthly meetings throughout 
the 6-month evaluation period. 

FINDINGS OF EVALUATION OR MONITORING 

The evaluation of independent living program 
by the CCS evaluation task force suggested that 
the program was providing good services to a very 
difficult and challenging population. Clients in the 
evaluation study demonstrated progress in 
self-estee.n, self-control, and decisionmaking 
skills and slight improvements in their interper- 
sonal and family relationships. Several problems 
were identified. Tney included the need for closer 
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transition period between group home residency 
and independent living. More definitive criteria 
for client acceptance were also seen as necessary 
in order to better screen potential clients. 

Overall, the program seemed to be operating 
well, but staff determined that in the absence of 
further GJC funding they would discontinue the 
independent living component in favor of 
establishing a stronger aftercare/transitional 
facility for girls leaving the residential group 
home component. 

RECOMMENDATIONS 

The CCS evaluation task force recommended: 
(1) a more structured screening process for 
clients, including specific acceptance criteria; (2) 
a transitional period of supervised living prior to 
placement in an apartment; (3) continuation of 
internal evaluation procedures using the instru- 
ments developed during the CCS evaluation: (4) 
refinements in recordkeeping to include specific 
short- and long-term client goals and monthly 
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review of progress toward these goals; and 
(5) continuation of supervised independent living 
for as many as six girls per year if alternative 
funding were availaoie. These recommendations 
were made to the program itself through the 
Governor's Justice Commission which had the 
authority to innplement the recommendations. 

STEPS TO ENSURE IMPLEMENTATION 

The CCS evaluation task force offered 
technical assistance and feedback to program 
staff throughout the evaluation process so that, at 
its conclusion, recommendations regarding 
internal evaluation, recordkeeping, screening, and 
providing a transitional period were already 
implemented. The GJC, as the funding authority, 
reviewed the task force's recommendations and 
met with program staff to review compliance. 
They also offered assistance in attempts to locate 
additional funding for program continuation. 
Short-term GJC funds became available and the 
program opted to strerigthen the transitional 
aftercare component with existing resources, 

EXTENT OF IMPLEMENTATION 

All recommendations of the CCS evaluation 
task force were implemented to the extent 
pojcible. 

SPECIAL BARRIERS OR SUPPORTS 

There were no particulai barriers to the 
evaluation. Major supports were the cotiperation 
of program staff, the active participation of 
citizen volunteers, and the strength of the 
evaluation techniques employed. Program staff 
were so pleased with the evaluation instruments 
snd prccedtires utilized by ^^f^ t'A<u^ fnrnft that 
they are now using them routinely to evaluate the 
effectiveness of their total program. 

RESOURCES AND COSTS 

Three citizen volunteers each donated approx- 
imately 20 hours of their time over the 6-month 
study period. Each task force member donated 
another 12-15 hours to the evaluation. The CCS 
evaluator put in approximately 100 hours for data 
analysis and report writing, as well as site visit 
supervision, at a cost of about $1,200. Clerical 
support amounted to another $300 and agency 
overhead and supplies added an additional $37!>, 
for a total evaluation cost of $1,870. The 
Governor's Justice Commission provided the funds 
for the evaluation. 

ADDITIONAL COMMENTS 

The task force's approach involves program 
staff in all stages of the evaluation and uses a 
formative/technical assistance style. As a result, 
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the evaluation of this program was a positive, 
interactive process for all involved. Recom- 
mended program changes were implemented 
quickly and without the need for coercion. The 
task force had the immediate sense that the 
program had benefited from the evaluation. This 
kind of experience makes tne citizen volunteers 
feel that their time has been well spent. 

Task force members, as a whole, felt that they 
had made a useful contribution in their roles as 
citizen volunteer evaluators. Most members saw 



their contributions as opportunities for offering 
substantive expertise in their professional fields. 
Additionally, they were aware that they brought 
an objective third-party perspective to the 
evaluation process. Members felt that they, tuo, 
had benefited from their experience on the task 
force. Major benefits they cited were a broader 
knowledge and awareness of social service 
programs in the State and an opportunity to learn 
and apply evaluation and research skills in a -real 
world** setting. 
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Scope or How Far Wffl You Cast the Net? 



Having assessed their organizational context and the extent of their 
involvement, Doug Brown and his committee are now ready to examine the 
scope of breadth of their evaluation tasVu There are several factors that 
influence this choice. First, Doug reminds the committee that the motiva- 
tion for the formation of the group in the first place was the association's 
concern about the lack of systematic county planning and the absence of 
realistic service and funding priorities. It is difficult to imagine how Doug's^ 
committee can satisfy this concern with an evaluation that focuses on only 
one program component. 

Second, Doug cautions the committee that their choice of scope must 
also be dictated by the level of resouc ces available to carry out any eval- 
uation or monitoring activity. Though, like most citizen groups, Doug's 
committee is composed of volunteers with lindted time, some of the mem- 
bers do have special expertise and access to other forms of assistance that 
will be valuable during an evaluation. For one thing, one of the committee 
members is an instructor in systems analysis at Calvin Coolidge Com- 
munity College. His skills vdll be enormously useful in processing and 
analyzing any data collected, and he also has access to student assistance. 

Another member of the board is a certified public accountant who is 
willing to assist in exploring cost and funding issues. Finally, Doug has se- 
cured the cooperation of the county mental health advisory board, which is 
willing to djare information and to provide other in-kind support. 

The decision regarding scope also is contingent vpon the extent of 
influence that citizen evaluators can expect to have. In this instance, the 
Greene County Mental Health Association's success in secijring expanded 
funding for local programs has eiisurei the cooperation of the coimty men- 
tal health department in the study tcid at least a sympathetic hearing of 
any recommendations. 

The combination of all these factors persuades Doug and his com- 
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political climate necessary to carry out a countywide stuffy of mental 
health programs. 



The cases highlighted in this chapter fall into 
four categories of descending magnitude or scope: 
statewide system, county or regional system, 
agencywide, and individual program component. 
Another passible category loader scope is "clus- 
ter.** Although no cases illustrating cluster are 
presented in this section, other cases, such as the 
Greater Chicago Mental Health Association Site 
Visitation Committee, represent this component 
of scope. In these cases, evaluators or monitors 
frequently review multiple facilities or programs. 

Sutewide Systems 

The first two cases, written by Lawrence Vel- 
asco and Bruce Braman, exemplify statewide 
system evaluation and monitoring. Velasco de- 
scribes the Colorado adult residential services 
monitoring project sponsored by the State devel- 



opmental disabilities division. This statewide 
project was initiated in 1978 because of the in- 
creasing size of the community residential system 
and the difficulties encountered by central office 
staff in monitoring multiple providers in such a 
large state. To cany out the monitoring, volun- 
teers were recruited from three regions. Infor- 
mation collected by the monitors during site visits 
was fed back to the State and \iltimately into the 
State licensing process. As a quality assurance 
process, the Colorado monitoring project proved 
very successful. As an ongoing quality assurance 
technique, however, it failed, since funding for 
the project was not continued. Though the scope 
of the project did not directly affect the citizens* 
ability to secure continued foanding, it does sug- 
gest that a smaller scale t Tfort would have been 
less costly and perhaps easitr to implement. 
The issue of funding is also a consideration in 



67 



ERLC 



76 



Bruce Braman's description of the Nevada rural 
clinics evaluation. Since the evaluation is man- 
dated by State law, however, the source of fund- 
ing for the effort is more stable. The rural clinics 
program Is State operated and provides a wide 
range of mental health services to 15 rural coun- 
ties and the Carson City area. In most areas, no 
other private mental health sendees are avail- 
able* State st2iff view the involvement of citizens 
as critical to continued public support for the 
clinics. Moreover, a grassroots effort that feeds 
into a statewide review of needs and service op- 
tions is essential in a State like Nevada. The 
geographic distances and time involved in travel 
have limited significant citizen participation ir 
some rural areas. 

County or Regional System 

The Sacramento County Mental Health Asso- 
ciation focused its activities on the seven agen- 
cies under contract with the Sacramento County 
mental health division. In this case, written by Al 
DeWcer, the countywide system was targeted be- 
cause of a recent change in program adminis- 
tration. Alteration in the patterns of service 
utilization following the change in county pro- 
gram auspices stimulated the association's con- 
cern and subsequent evaluation. The association's 
focus on the county system was especially im- 
portant given the change in administrative aus- 
pices. Even thor^ the association's analysis and 
findings were critical to county operations, its 
recommendations v/ere internal to the association 
and therefore of limited influence in the county. 

Agenpywide 

Ralph Denty, from the Brunswick, Georgia, 
area, descrlucs uuw Uie mental health association 
in that community carried out an agencywide 
evaluation in partnership with the local mental 
health advisory council. The target of the evalu- 
ation was the service delivery system of the 
Coastal Area Community Mental Health Center. 
Secondarily, this case also exemplifies an evalu- 
ation of a regional service system since the cen- 
ter's catchment area . 3vered seven counties. The 
collaboration heVueen the mental health asso- 
ciation and the local advisory council allowed the 
evaluators to extend the target of the evaluation 
to Lnclude tho entire catchment area. As a result 
of the comprehensiveness of the re^^-^w activity, 
many problem areas were identified and resolved. 

Joan Wooley and Dorothy Kurjan describe 
mother agencywide evaluation, one that focuses 
on^ the delivery of emergency shelter services to 
children and adolescents in the western region of 



Massachusetts. In the spring of 1982, members of 
a children's committee, mandated by State law tc 
review and assess children's services, conducted 
an evaluation of The Shelter, an emergency 
shelter for adolescents. The focus on one agency's 
service system permitted committee members to 
address in-depth specific concerns regarding the 
ongoing stability and quality of services being 
provided by the program contractor. As a result 
of their investigation and subsequent recom- 
mendations, the existing vendor was changed, and 
other administrative changes were mrjde. 

Program Component 

In the final subsection of thirr chapter, two 
different cases exemplifying evaluation of a sin- 
gle program component are described. Elizabeth 
Fulton and Bruce Hirsch, in case #27, write about 
the San Francisco Mental Health Association's 
(MHA) preliminary evaluation of psychiatric 
emergency services provided in two San Francisco 
hospitals. Members of the MHA*s public affairs 
committee selected psychiatric emergency serv- 
ices as their focus since they function as the hub 
of conununity mental health services in the San 
Francisco area. Eight MHA volunteers organized 
and implemented the evaluation with the as- 
sistance of the directors of the psychiatric emer- 
gency services units. By targeting their evaluation 
to a specific program component, association 
volunteers were able to obtain ad/2tional knowl- 
edge and experience in a critical service area, 
while gaimng credibility in the eyes of the pro- 
fessional medical community for their interest in, 
and commitment to, improving emergency serv- 
ices. The initial results of this evaluation pointed 
to the need for another study to identify the 
availability and appropriateness of referrals for 
psychiatric emergency services. 

In case #26, Jan Foutz an^l Barbara Goza 
higihligiht the Salt Lake City, Ut^^i, Granite Com- 
munity Mental Health Center Advisory Council's 
evaluation of the adult outpatient services group 
orientation procedure. In this case, both center 
staff and advisory council members recognized 
the need for a new orientation process but were 
cor. emed about potential implementation prob- 
lems. The scope of this evaluation, therefore, 
focuses on one aspect of one sendee-delivery unit 
in one CMHC. After conducting an extensive re- 
search effort that included analyzing complaint 
files and client fiow data, members of the advi- 
sory council were successful in implementing 
many of their recommendations. One ot the key 
factors in their success was the early involvement 
of 'Jie unit staff and management in the evalua- 
tion process. 
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21. A Statewide System for Monitoring Community 

Residential Services 

Lawrence A. Velasco 

Pueblo County Board for Developmental Disabilities, Inc. 



SUMMARY 

The Pueblo County Board for Developmental 
Disabilities, Inc, participated in the development 
and implementation of a statewide System for the 
Monitoring of Community Residential Services 
beginning in the summer of 1978. This monitoring 
system is the product of the Colorado Division for 
Developmental Disabilities' residential section. 
The objective of this unique system is to provide 
an annual onsite monitoring and evaluation of all 
community -based adult residential facilities 
licensed by the State of Colorado and funded 
through the division for developmental disabilities. 

TYPE OF ORGANIZATION 

The system for monitoring of community 
residential services was organized by the 
residential system of the Colorado Division for 
Developmental Disabilities. Because of the large 
geographic size of the State (I04,2/)7 square 
miles) and because of the number of residential 
facilities being developed (68 community 
facilities) it wa^ imperative to initiate an 
accountability system that would insure quality 
control of all physical facilities and client 
programs being planned or provided throughout 
the State. A team concept was utilized to 
formulate the nucleus of the system. 

Each team was composed of persons repre- 
senting one of the following areas: The State 
division for developmental disabilities, an 
association for retarded ::itizens, a coT.r-.unity- 
centered board, the developmental disabilities 
council, and one of the three State home and 
training schools. 

EVALUATORS OR MONIT ORS 

The State was dissected into three region?. 
Each region has persons on a list of eligible 
evaluators who have volunteered to take part in 
this exercise. These individuals have professional, 
paraprofessional, or consumer representative 
backgrounds. All persons are trained in the 
process of the evaluation by a member of the 
residential section of the division for develop- 
mental disabilities. The training session covers 
the philosophy, rationale, evaluation checklist. 



For further information vjrite Judy Wood, 210 Fairvicw, 
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and rating procedures to be used. The division 
residential section selected regional team leaders 
who are responsible for assigning the volunteers 
from their region for site visits. The schedule for 
evaluations is made in October and all evaluations 
are completed during the winter and spring. 

REASONS FOR THE EVALUATION 
OR MONITORING 

Colorado launched into the process of dein- 
stitutionalization after the completion of a 
statewide study entitled The Environmental 
Design Group Study which was completed in July 
1978. The study identified the inadequacies of the 
three State institutions and identified large 
numbers of individuals who could be better served 
in the community in small eight-bed facilities. 
Since the division for developmental disabilities 
had been contracting since 1965 with 22 com- 
munity-centered boards in the State to provide 
day program services to developmentally disabled 
clients, it seemed highly appropriate to expand 
those community services and some existing group 
homes to include institutionalized individuals who 
could profit from a community rasidential 
environment. As the number of group homes 
oegan to increase in 1978, the division for 
developmental disabilities recognized the need for 
instituting a monitoring system that wouid iruubll 
the practice of "warehousing" in the community. 
A. ^ough the Colorado State Legislature and the 
executive branch of government were completeily 
in favor of the deinstitutionalization process, 
monies had not been appropriated for the 
monitoring portion of these services. Therefore, 
the State developmental disabilities council 
funded a grant of approximately $4,200 to develop 
and implement a process of evaluation and 
monitoring of adult residential facilities through- 
out the state. 

LEVEL OF PARTICIPATION 

The Pueblo County Board for Developmental 
Disabilities, along with several other com- 
munity-centered boards, was asked to assist with 
the developntent of a monitoring and evaluation 
tool by the State division staff. The Pueblo board 
had been included because it had successfully 
provided childrens' residential services since 
1970. It had decentralized its large community 
facility of 42 beds to 6 bilevel homes spread 
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throughout the community, thereby establishing 
itself as an authority in small group homes. All 
participating communUy center boards provided 
items for the checklist and suggestions throughout 
the process of evaluation, 

TARGET OF EVALUATION 
OR MONITORING 

The first site visits took place in the fall of 
1978, The objective was to review 68 residential 
facilities by June 1979. The teams reviewed 
facilities in Boone, Alamosa, Salida, and Colorado 
Springs. One full day was allotted for travel to 
and from each site and for the administration of 
the evaluation. Travel time ranged from 30 
minutes to 2 hours. The Boone Guest Home is 
located appz-oximately 20 miles east of Pueblo in 
a small fairning community of approximately 200 
people. The facility is a ""mom and pop" operation 
and is located in a renovated boarding home. This 
facility is one of the larger group homes and is 
licensed for 15 adults. Each adult has his/her own 
separate bedroom and all residents share a 
common recreation/living room and dining room. 
The Alamosa Group Home, which houses eight 
clients, is located about five blocks from the 
downtown area. Alamosa is a community of about 
30,000 and is located in the San Luis Valley 
approximately 80 miles south of Pueblo. The 
Salida Group Home is located approximately 100 
miles west of Pueblo and is in a community of 
about 10,000 people. The Colorado Springs facility 
is located in the mountains and is called Cheyenne 
Village. There are 4 cabins that house 10 persons 
each. Colorado Springs is located approximately 
50 miles north of Pueblo and has a population of 
300,000. The Pueblo County Board operates four 
adult group homes in the city and county of 

n..-^t«io« n..-»ui^ . ^^m.. .t ^Li^^ ^£ »«>r rw>/\ „_j 
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located approximately 1 10 miles southeast of 
Denver, Colorado. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The checklist for monitoring community 
residential services focuses on a number of areas: 

(1) physical setting (location in the community); 

(2) accessibility to the community (recreation, 
shopping); (3) interior design (normal environment, 
personal space, and group activity space); 

(4) private areas (design and furnishing of 
bedrooms); (5) resident programs and services 
(food, git)oming, attention to learning, community 
living); and (6) administration (program manage- 
ment, general support services, management of 
residents* rights, dignity, safety, and resident 
movement to less restrictive living). 

TECHNIQUES USED 
A checklist with 71 items was used by each 
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member of the team. Each person was responsible 
for completing his/her own ratings. Prior to the 
administration of the evaluation, the team leader 
introduced team members to the staff and any 
residents present. The team leader reviewed the 
survey process with everyone, including the 
purpose and procedure for the postvisit team 
conference. Th'^ team was divided into two 
groups. Group A interviewed direct care staff 
with particular attention to section 2 of the 
checklist. Group B interviewed administrative 
staff and attended to checklist section 3. The 
process takes approximately 4 to 5 hours because 
of the number of questions that need to be 
addressed. 

FINDINGS OF EVALUATION OR MONITORING 

Our agency staff have benefited greatly from 
this evaluation process because of the amount of 
sharing between team surveyors and staff persons 
answering questions about our programs. Our 
experience is that every group home has an 
individual personality and is unique with respect 
to the operators, decor, furniture arrangements, 
meal preparations, leisure time activities, 
community involvement, etc. One universal 
problem affecting the group homes is the training 
of facility staff. Because of the funding level, 
most programs have only a minimal orientation to 
residential services and developmental disabil- 
ities. It is apparent that the State must try to 
secure funding for additional courses in develop- 
mental disabilities, program development, 
nornrialization theory, client assessment, and 
provide a continuum of learning sequences. 
Generally speaking, however, the programs are 
effective and the residents appear to be happy. 
The process of monitoring and evaluation does 
ctHHtscJi lu piety cr Icjrye pari in enhancing agency 
pride. Everyone wants to bp a part of a winning 
team. 

RECOMMENDATIONS 

After the onsite evaluation, the team recon- 
venes to complete the consolidated checklist 
answer sheet. The team reviews each item and 
secures a consensus on the rating. Recommenda- 
tions are made by the team members in order to 
assist the group home operator in improving 
his/her services. Ideas are given and are recorded 
on the evaluation sheet. The division for develop- 
mental disabilities staff member takes this 
information back to the central office and sends 
out an official evaluation with rotings and 
recom mendations. 

STEPS TO ENSURE IMPLEMENTATION 

At the end of the evaluation, an exit con- 
ference is held with the facility operator. The full 
checklist is reviewed and recommendations for 
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improvement are discussed. The facility ooerator 
is also praised for the positive aspects of the 
programs and facilities. Within 2 weeks, the 
operator receives the official evaluation and is 
given approximately 1 month to provide a plan of 
correction for the items that are in need of 
remedial action. In the plan correction, the 
operator must identify the dates by which all 
deficiencies will be corrected. If any items are of 
great significance, division staff return to review 
the required changes. It has been our experience 
generally that everyone is willing to cooperate 
with the changes as required. One of the reaoons 
for such cooperative attitudes is believed to be 
the peer review procedures. Because their ideas 
are sought in the evaluation process, the opera- 
tors do not feel as though they are being attacked 
or pressured. Over the 4 years of evaluations, a 
measurable level of improvement has been seen in 
the system jf adult residential services. 

EXTENT OF IMPLEMENTATION 

This system of monitoring and evaluating adult 
residential services in Colorado must be labeled a 
magnificent success. One hundred percent of all 
adult residential facilities licensed to serve 
developmentally disabled in Colorado are 
evaluated annually. All facilities are required to 
participate as part of the criteria for funding ana 
licensing. This process has brought all operators 
under the umbrella agency of the division for 
developmental disabilities and has enhanced the 
collection of data in all categories of operation. 
This information, coupled with quality control, 
has provided the Colorado State Legislature and 
the executive branch of government with the 
verifiable assurances of the staff of the division 
for developmental disabilities and the State 



developmental disabilities council that adult 
community residential services are a viable 
service delivery system. 

SPECIAL BARRIERS OR SUPPORTS 

From the very beginning, there was a big 
response from the field to volunteer for the 
evaluation process. The only expense that the 
division incurred was the cost of travel and per 
diems. However, because Federal and State 
monies are becoming tighter, these funds may not 
be available in the future. The system, we hope, 
will continue to provide the volunteers to insure 
that the qtilOffty control continues even though it 
may cost each agency and individual a few dollars. 

RESOURCES / '^' D COSTS 

During the first vear of the evaluations, the 
division scheduled 150 volunteers and evaluated 
68 group homes. In 1981, 150 volunteers evaluated 
76 grDup homes. Currently there are 122 facilities 
in the State and this number is expected to 
increase to 150 facilities by September 1982. The 
division for developmental disabilities is trying to 
secure approximately $6,000 for the travel and 
per diem expe."5S9S of about 200 \olunte ts. 

ADDITIONAL COMMENTS 

Generally, ail community center boards in 
Colorado have been supportive of this system of 
evaluation and monitoring. It has become an 
accepted part of our service delivery lives. The 
process is being replicated in another area of 
service delivery in our State system — transpor- 
tation services. It is anticipated that this ntw 
venture will meet with equal success. 



22. Citizen Groups Evaluate a Statewide Mental Health 

System in NevaJa 

The Reverend Bruce O. Braman 

Ad)nsory Council for Rural Clinics Comntunity Mental Health Center 



SUMMARY 

Citizens play an active role \n oroviding 
guidance and evaluation for the Rural Clinics 
Community Mental Health Ce iter (RCCMHC). 
Local groups, be they mental nealth advisory 
boards or mental health asso^iations, provide 
ongoing feedback to the loc'*l satellite clinics and 
their staff. An advisory council represents the 
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collective views of citizens in the rural counties 
of Nevada and communicates these views to the 
central administrative s^aff. This case describes 
one of the periodic ev. luations that utilizes two 
levels of citizen involvement. The local advisory 
boards conduct community forums that allow 
citizens to express their views on mental health 
isfues and needs and on tne quality of service 
provided by the satellite office. The boards also 
administer questionnaires designed by the central 
administrative office that generate specific and 
valuc^le data. The second level of citizen 
involvement is the advisory ccuncil. The system- 
wide advisory council participates with repre- 
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sentatives from the satellite offices, their 
managers, and central administrative staff in an 
annual evaluation session. The previous year's 
objectives and operations are critically assessed. 
Data from satellite areas are reviewed and a set 
of recommendations is produced to serve as a 
basis for an action plan for the Rural Clinics 
CMHC during the coming year. 

TYPE OF ORGANIZATION 

The local advisory boards may take varying 
shapes. In some areas, the county mental health 
association agrees to provide the citizen advisory 
function. In other areas, the advisory board is 
linked directly to the Rural Clinics CMHC. In 
either case, the citizens themselves tend to 
generate membership from among their friends, 
professional contacts, and community leaders. In 
a few places, satellite office managers may have 
varying roles in constituting a boaid's membership. 

EVALUATORS OR MONITORS 

The evaluation is carried out at the local level 
by advisory board members in conjunction with 
the advisory board member representing that 
catchment area. The makeup of the local boards 
varies from place to place, but typically includes: 
professionals involved in providing human serv- 
ices, government officials, representatives from 
business, and citizens with an interest in mental 
health issues. Their varied interests and wide 
assortment of contacts provide a good mix for 
generating helpful data. Staff members in the 
Central Office of the Rural Clinics CMHC 
provide evaluation data and access to clients and 
former clients. 

The annual evaluation session attempts to 
synthesize the information collected at the local 
level and turn it into manageable reconirnenda- 
tions. At this point, care is taken to include three 
persons from each catchment area: the satellite 
office manager, one professional staff member 
(psychologists, social workers, etc.), and the 
advisory council member. The central staff is 
available for consultation and guidance. They also 
represent policies from state offices that have a 
bearing on the issues under study. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The evaluation is mandated by State law, but 
this fact does not really tell the story. The 
central staff encourage and solicit the evaluation 
and have consistently regarded citiaen involve- 
ment in the process as critiuully -riportant. The 
evaluation is conducted annually and its results 
are reviewed by the State Division of Mental 
Hygiene and Mental Retardation. Program 
policies are established as the result of recom- 
menoations that emerge in the evaluation process. 
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The evaluation also serves a more crucial 
function; it is an invaluable source of support to 
the rural clinics program when programs and 
budgets are presented to the State legislature. 

LEVEL OF PARTICIPATION 

This author has been able to experience the 
evaluation process from several vantage points. 
Typically, the identification of areas to be 
evaluated are designated by central staff. At 
some points, however, the advisory council has, on 
its own, carried the evaluation into other areas. 
At the locpl level, it has been interesting to 
collect data both formally (community forums, 
questionnaires) and informally (such as interviews 
with office managers and staff of the local 
center). At the area level, both the advisory 
councirs evaluation of data and the large annual 
evaluation session have shown the agency's ability 
and willingness to respond to the evaluation 
recommendations both openly and creatively. 

TARGET OF EVALUATION 
OR MONITORING 

Rural Clinics Community Mental Health 
Center, an ogency of the State of Nevada, has a 
mandate to provide a wide range of mental health 
services to the rural counties of the State. The 
area covers 15 counties and the Carson City area. 
Only the two highly populated counties, Washoe 
and Clark, are excluded from its service area. The 
agency covers an area of 96,000 square miles and 
a population of 154,000 or 1.6 persons per square 
mile. The communities served are typically small 
cities or towns. The author's own town has a 
population of 12,000 and the local mental health 
center serves a county the size of New Jersey. 
This is typical. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

In this evaluation, attention was given to all 
aspects of the current services of RCCMHC. In 
later stages of the evaluation, priorities were 
assessed and certain services were deemphasized 
in agency policies. Special attention was given to 
community needs that were not being met. The 
evaluation also covered the clinics' management 
information systems, the future roles of the 
center's advisory board and council, central office 
functioning, and the development of community 
support strategies. 

TECHNIQUES USED 

The following methods were used to gather and 
process information: 

1. Preliminary sessions at the local level — these 
sessions focused on specific 
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issues and needs that were relevanl to the 
local areas. They furnished hard data from 
the local level through questionnaires and the 
management information system. 

2* Evaluation sessions at the advisory council 
level — members of the advisory council 
participated in one of six task forces^ each of 
which addressed a different issue. (The issue 
differences included: the management 
information system, the role of boards and 
the council, strategies for developing 
priorities, new program development, 
political action, and the structure of rural 
clinics.) Task force groups assessed the 
feelings of board(s) and staff in regard to 
each issue. To do this, each group used a 
three-part process. The first step was to 
define the '•idear without consideration of 
any constraints. The second step was to 
identify constraints and limitations. The final 
step was to make specific recommendations 
in light of the ideal spelled out in the first 
step, and the constraints spelled out in the 
second step. The recommendations of each 
task force were then reviewed in a general 
session of the advisory council. 

3. Review of evaluation results by the central 
office — staff from the central office 
reviewed all task force recommendations, 
assigned tasks to the central office staff, and 
formulated an agency plan for the coming 
year. 

FINDINGS OF EVALUATION 
OR MONITORING 

Analysis of data from the local service areas 
suggested that there was a general need for 
alcohol and drug abuse services, services for 
high-risk families and children, and residential 
treatment programs for adolescents. 

RECOMMENDATIONS 

Each of the six task forces of the advisory 
board compiled a series of recommendations. 
Some of the recommendations included: 

• ClarifVing the council's role 

• Establishing community committees 

• Establishing a residential treatment program 
for adolescents 

• Appointing a task force to develop a training 
packet for paraprofessionals 

• Providing a greater opportunity for direct 
communication among satellite office 
managers 
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STEPS TO ENSURE IMPLEMENTATION 

Responsibility for implementation rests with 
the center director. However, the advisory 
council continues to monitor implementation in 
its watchdog capacity. It has, from time to time, 
expressed concern over specific matters to the 
director of RCCMHC. 

EXTENT OF IMPLEMENTATION 

Implementation has been built into the 
evaluation process. The annual plan cannot be 
finished until the recommendations arising out of 
the annual evaluation meeting are received. After 
the central office reviews the evaluation data, 
many changes, particularly in the ordering of 
priorities, are made and plans for major inno- 
vations commence. But when major funding 
changes are called for, the process tends to be 
lengthy. Ultimately, some programs, such as the 
rural treatment center for troubled adolescents, 
must await legislative action and probably a 
change in the current economic climate. 

SPECIAL BARRIERS OR SUPPORTS 

The distances between population centers are a 
barrier to major citizen participation in rural 
areas. Communication between areas has been 
limited to representation of each area on the 
advisory rouncil and to communications with the 
central office. Also, time limitations have 
constrained activities in some areas. But evidence 
of support for the evaluation was the high citizen 
commitment to RCCMHC; its services are not 
paralleled by the private sector. As a result, the 
citizen interest is real and, with diminishing State 
funds, the interest is accompanied by a sense of 
urgency. The staff of RCCMHC view the citizens 
of Nevada's rural communities as the primary 
support for the continued delivery of publicly 
funded mental health services. Without the 
willingness of citizens to serve as advocates for 
mental health services throughout Nevada's rural 
counties, the program would quickly falter. For 
this reason, the center staff accept the impor- 
tance of an evaluation model that relies on 
citizen participation. Satisfaction, dissatisfaction, 
and desires are received with all seriousness. 
Also, citizen involvement has served to intensify 
the level of commitment to the center. On the 
other hand, citizens' participation and impact is 
appropriately diffused so that final recommen- 
dations reflect a collective consensus rather than 
the views of a few individuals who may be 
outspoken. 

RESOURCES AND COSTS 

Because evaluation is an ongoing part of the 
rural clinics' life of RCCHMC, the expenditure of 
human resources and money iz very hard to gauge. 
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iir annual evaluation session involved 23 
stai I members and 6 citizens for 2 workdays at an 
approximate cost of $5,500. Time and cost 
estinnates are not available for the local advisory 
board sessions. T.,e advisory council conducts its 
business in four sessions per year at a budget cost 
of $4,000. The six members spend a minimum of 1 
full workday per session. 

ADDITIONAL COMMENTS 

Speaking as one of the citizens involved in the 
evaluation process, the a ithor is heartened by the 
lasting impact that the evaluation process has had 



on the operations of Hural Clinics Community 
Mental Health Centers. The time spent has 
produced positive changes and has served to help 
assure the ongoing presence of mental health 
services for the citizens of Nevada. The absence 
of any adversary relationship in this model and 
the mutual regard and cooperative spirit in the 
review is a spirit that should be nurtured carefully 
in undertaking any similar venture. 



23. Mental Health Association Evaluation of County 
Mental Health Services 

Ai Dekker 

Mental Health Association, Sacramento Chapter 



SUMMARY 

In early 198U - subcommittee of the public 
affairs committee of the Mental Health Asso- 
ciation, Sacramento Chapter fMHASC) made eight 
site visits to seven agencies representative of 
nnental health services in Sacramento County. The 
purpose of the visits was to assess problems and 
constraints in the system administered under 
contracts with the county mental health division. 
The subcommittee examined programs, rehabi- 
litation activities, health and safety, accommoda- 
tions, menus, and accessibility of the system. The 
results were compiled In a report that described 
what was being done particularly well, listed the 
problems that were identified, and recommended 
specific actions by MHASC. 

TYPE OF ORGANIZATION 

The public affairs committee of the MHASC is 
an advocacy group composed of members of the 
board of directors and other volunteers from the 
association membership. The chairperson is 
chosen by the president of the board. The site 
visit subcommittee was a volunteer group whose 
chairperson was appointed by the chairperson of 
the public affairs committee. The public affairs 
committee advises the board on public stands to 
be taken on mental health issues. 

E VALUATORS OR MONITORS 

The four members of the subcom.mittee 
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conducting the evaluation were: an accountant, a 
former school teacher, a licensed vocational 
nur I, and a retired research and development 
manager. 

REASONS FOR THE EVALUATION 
OR MONITORING 

Piior to July 1, 1979, the University of 
California at Davis administered the mental 
health system for Sacramento County. After that 
date, the system was administered by the county 
mental health division through contracts with a 
number of providers. Data analysis for the 6 
months following this transition showed that the 
average outpatient count decreased by J O percent 
in comparison with the county for the preceding 6 
months. This stimulated interest in assessing the 
problems and constraints of providing care in the 
new county system as a basis for focusing future 
efforts of MHASC. 

LEVEL OF PAKTICIPATION 

The outpatient counts were obtained from the 
county mental health division and analyzed by a 
subcommittee member. Members of the subcom- 
mittee selected representative sites to visit and 
checklists were prepared. Two to four members 
participated in each visit. The subcommittee 
chairman prepared a summary report. 

TARGET OF EVALUATION OR MONITORING 

This study focused on seven agencies under 
contract with the county mental health division. 
The county has a population of 783,000. The 
agencies selected covered a broad spectrum of 
care: 
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• A crisis emergency center 

• A neighborhood mental health center 

• A children's inpatient unit at a private 
general hospital 

• A children's outpatient unit at the same 
hospital 

• An experimental program treating clients in 
an outpatient mode who are so disruptive 
that they are usually treated in a more 
expensive inpatient nnode 

• A residential care facility 

• An integrated mental health rehabilitation 
program built on the social model providing 
services for those who require ongoing 
mental health treatment 

• A board and care home (not under contract 
with the county) 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The issues addressed were the degree to which 
agencies adhered to the requirements of Califor- 
nia J w and the service delivery problems and 
constraints identified by agency supervisors or 
staff. 

TECHNIQUES USED 

The techniques used included site visits 
supplemented by a checklist for facility review. 
The checklist was prepared by tha full subcom- 
nnittee and covered the relevant requirements of 
the California Code. One side of a legal size sheet 
of paper contained information obtained in a prior 
telephone inquiry. The other side contained 
specific suggestions for observations to be made 
during the site visit. Supervisors and staff were 
interviewed during the visit. Discussions were 
held with clients at only one facility. At this site, 
a consumer*satisfaction survey was conducted 
with 46 clients. 

FINDINGS OF EVALUATION OR MONITORING 

MoniVjring the number of ou tpatient visits in 
the entire system during ^he Vscal year 1980-81 
showed no significant further change from the 
30-percent decrease in visits that had occurred 
during the 6 ninths following the transition in 
administrations. Each of the agencies appeared 
to be clean, well-organized, and to be doing a 
creditable job. One was doirig an outstanding job 
in publicizing its activities under the new system 
and had rebuilt the client load. Examples of 
problems identified by the interviewees are: 
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• Preventive mental health is not receiving 
sufficient attention in the schools. 

• Children's psychiatric emergency services 
are unavailable. 

c Transportation of children in daycare 
services is difficult to arrange. 

• Peer counseling is lacking in junior nigh sr^hooL 

• The crisis emergency center has no way of 
segregating disruptive clients in the waiting 
area and is unable to handle nonambulatory 
clients and clients under 18 years of age. 

• Bus routes and times of operation are 
inconvenient for clients attending the 
neighborhood mental health center and bus 
passes cannot be bought in the neighborhood. 

• More facilities are needed to bridge the gap 
between board and care homes and com- 
pletely independent living. 

• There is a shortage of counselors for those 
residences in which clients prepare for 
independent living. 

• The**e is a shortage of jobs for clients. 

• The California State hospital system treats 
only those clients with good prospects for 
rehabilitation (this means that the county 
must treat the long-term chronically 111, 
some of who are so agitated that they need 
to be under permanent care In Institutions). 

• The consumer satisfaction survey (conducted 
at only one facility) was remarkable for the 
rarity of strongly negative comments. 

RECOf^ENDATIONS 

Recommendations were directed to the public 
affairs committee of the MHASC which endorsed 
the report and recommendations, and directed 
them to the MHASC board of directors, which in 
turn adopted them. The recommendations are 
listed In order of priority: 

• Sample consumer opinion concerning 
problems and needed improvements in the 
mental health systerr. 

• Work to obtain employment opportunities for 
mental health consumers returning to the 
community. 

• Determine the extent of peer counseling in 
elementary, junior high, and high schools In 
the county and lobby for its inclusion wher*= 
absent. 
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• Investigate whether preventive mental health 
programs exist in the schools of the county 
and if so. their quality, and lobby for their 
inclusion or improvement as necessary. 

• Persuade school systems to develop teachers' 
awareness regarding resources available for 
handling family and other mental health 
problems. 

• Assess the availability of children's psy- 
chiatric emergency services, examine various 
options for improving services, and lobby for 
selected approaches. 

• Become familiar with client and proviaer 
problems of the board and care homes 
through discussions with agency represen- 
tatives, case managers, and State licensing 
officials. 

• Work to initiate a patient tracking system 
that insures continuity of care without 
infringing on clients' rights to privacy. 

• Continue site visits and attempt to develop a 
relationship with the mental health advisory 
board that would involve joint MHASC and 
advisory board site visits (the mental health 
advisory board is a statutory advisory board 
that reports to the county bjard of super- 
visors, and is separate from the MHASC). 

• Assess the extent to which the need for 
rehabilitation programs for long-term 
consumers of mental health services is being 
met and lobby, if necessary, for expanding 
these services. 

• Study alternative ways of dealing with 
long-term, chronically ill clients who are so 
agitated that they need to be under perma- 
nent care in institutions, select approach, and 
lobby for it. 

STEPS TO ENSURE IMPLEMENTATION 

Recommendations were internal to the 
association. 

EXTENT OF IMPLEMENTATION 

The association has initiated a preventive 



mental health program in elementary schools. The 
program is titled the Tm Thumbody** program, 
and is directed toward building self-esteem in 
second-grado students. Sampling of consumer 
opinion has been initiated at an additional 
facility. The county mental health division has 
developed a computer-controlled billing system 
that is the first element of a planned client- 
tracking system. 

The recommendations for MHASC action were 
so extensive tnat only a few could be addressed 
with the limited number of volunteers. They do, 
however, provide an agenda for future work. 

SPECIAL BARRIERS OR SUPPORTS 

The only significant barrier occurred when an 
attempt was maae to visit a board and care home 
that had been assessed fines by the State for 
violations of regulations. Because the proprietor 
did not permit the State visit, the subcommittee 
substituted a board and care facility whose 
owner/manager was acquainted with a member of 
the MHASC board of directors. 

The subcommittee members were very 
important to the success of the project. They 
were very interested, worked hard, and, as a 
result of their inquisitiveness, a great deal was 
learned. 

RESOURCES AND COSTS 

A conservative estimate of the total time spent 
by the four volunteer members of the subcom- 
mittee is 175 hours. Very little staff time was 
involved. The subcommittee typed checklists and 
reports, but the staff reproduced them. Trans- 
portation was paid by the subcommittee members. 

ADDITIONAL COMMENTS 

The most important thing learned was that 
talking with management or supervisors and tours 
of the facilities are of limited value in disclosing 
some of the serious problems that can exist. Tho 
contract for one facility that gave us a very 
favorable impression was subsequently terminated 
for poor management and allegations that several 
clients were sexually assaulted by employees. 
Next time, it would he important to find a way to 
talk with clients, former clients, and/or families 
of clients. Perhaps 75 percent of the time should 
be devoted to this activity. 
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24. A Site Visitation Tean:'s Assessment of a Coastal CMHC 

Ralph E. Denty, Jr. 

Area 34 Mental Health Advisory Council and Glynn County 
Mental Health Association 



SUMMARY 

During l97'*-78, the Glynn County Mental 
Health Association (MHAGC) and the Area 34 
Mental Health Advisory Council (MHAC) imple- 
mented a project to assess and evaluate the 
service delivery system of the Coastal Area 
Community Mental Health Center (CACMHC) 
which provides services to the Counties of Bryan* 
Camden, Glynn, Liberty, Long, and Mcintosh in 
Georgia. Site visits were made to the center and 
interviews were conducted with center staff, 
selected public agency personnel and other 
persons in the catchment area. These interviews 
and visits resulted in the development of a formal 
report that summarized the team*s findings and 
recommendations. 



director, and the executive secretary of the 
MHAGC. 

REASONS FOR THE EVALUATION 
OR MONITORING 

As an advocacy group, MHAGC felt a respon- 
sibility to determine if the services provided by 
the CACMHC were adequate to meet the mental 
health needs of the citizens in the catchment 
area. One of the mental health association goals 
is to evaluate periodically the services provided 
by the center. It was also felt that the advisory 
council should be included as a participant in the 
evaluation. 

LEVEL OF PARTICIPATION 



TYPE OF ORGANIZATION 

The Mental Health Advisory Council (MHAC) 
consists of 30 members drawn from the counties 
served by the CACMHC. The members are 
appointed by the director of the Georgia Division 
of Mental Health after nomination by the local 
health officer and the approval of the State 
Mental Health and Mental Retardation Advisory 
CounciL The MHAC, whose composition complies 
with State standards for age. sex, place of 
residence, and income level, meets bimonthly and 
is responsible for approving the CACMHC's 
budget, hours of service, and selection of center 
director. The Glynn County Mental Health 
Association (MHAGC) is composed of interested 
citizens who, through their support, leadership, 
and advocacy, develop programs and disseminate 
information to promote mental health in the 
community and to ensure that adequate services 
are provided for the mentally ilU 

EVALUATOR S OR MONITORS 

This evaluation was conducted by twelve 
volunteers from the Glenn County Mental Health 
Association (MHAGC) and MHAC. These indivi- 
duals included a county commissioner, two 
ministers, a college professor, a school teacher, a 
doctor, a social service worker, a funeral 



For further information write Ralph E. Denty, Jr. 
Ph.D., Brunswick Jimior College. Altama at Foiirth, 
Brunswick, GA 31523, (912) 264-«7235. 



Project members were involved in every stage 
of the evaluation— from planning to assessment. 
Members were trained and equipped with the 
information, skills, and attitudes needed to 
conduct the study. All interviews, data collection, 
data analyses, and report writing were conducted 
by the members. When requested, secretarial and 
professional assistance were provided by the 
center* 

TARGET OF EVALUATIOISI OR MONITORING 

The purpose of the site visitation project was to 
evaluate the delivery of services by a regional 
mental health center to a six-county, coastal area 
of Georgia. Services provided by the center 
include outpatient, day treatment, emergency, 
consultation and education, drug abuse, alcoho- 
lism, screening, foUowup, child and adolescent 
treatment and supportive living. A satellite 
progran in Liberty County provides services for 
outpatients, children, adolescents, and drug and 
alcohol clients. Training centers for mentally 
retarded persons are located in four counties, and 
two counties have an outreach program of 
supportive counseling and foUowup medication for 
outpatients. 

The center has a catchment area population of 
104,000 and a median family income of $7,289 
with 19.1 percent of the families receiving a 
yearly income of less than $6,000. Approximately 
2,325 families (8.2 percent) are on public 
assistance. Administratively, the center is 
responsible to the local health officer and the 
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Glynn County board of health, which is the 
contracting agency for the center's services. The 
health officer, in turn, is responsible to the 
directors of the Georgia mental health division 
and the physical health division. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The major areas assessed by the site visitation 
team included the following: the quality of 
communication between the CACMHC and the 
agencies and citizens served: the continuity of 
care; the degree of citizen involvement in onging 
policy determination of center programs; the 
implementation of all minimum required services; 
the time frame and procedures for adding new 
services: and the accessibility of services 
provided by the center. 

TECHNIQUES USED 

Information for this evaluation was obtained 
through site visits to the CACMHC and interviews 
with center staff. Interviews were also conducted 
with a wide range of key informants, i.e., 
knowledgeable community individuals including 
school principals counselors, regular and special 
education teachers. Junior College students, 
policy department personnel, judicial members, 
lawyers, public service agency staff, physicians, a 
director of the emergency room at the local 
hospital and a random sample of citizens from the 
catchment area. The Working Manual of Simple 
Evaluation Progra»n Techniques for Community 
Mental Health Centers, published by the U.S. 
Department of Health and Human Services 
(DHHS) in 1976» was used as a guide in conducf 'ng 
this research; the survey instrument contained in 
the manual served as a basis for the interviews. 

FINDINGS OF EVALUATION 
OR MONITORING 

The site visitation team was impressed by the 
high quality of services offered by the CACMHC 
and by the staff's commitment to their clients* It 
was discovered, however, that more publicity and 
better public relations were needed in order for 
the community to become more aware of the 
center, its location, and services. Those who were 
familiar with the center felt that the services 
provided were good and should be continued; 
however, the need for better public relations was 
also expressed. In counties other than Glynn, 
communication between the center and the public 
was even more limited. The center appeared to be 
best utilized in providing services to students with 
learning disabilities. 

The team found that center staff put forth a 
great deal of effort to make sure that clients 
received continuity of care. Despite this effort, 
when patients were released from the regional 
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mental hospital in Savannah, there appeared to be 
a breakdown in communications between the 
State hospital and the center. Moreover, partial 
hospitalization for mentally disturbed patients 
was not available in the local hospital. In addition, 
certain geriatric services and a formalized 
coRS'Jltation and education program were not 
available at the center. In Liberty County the 
services were also limited, and theie appeared to 
be some operational oroblems in providing 
services for Fort Stewart military personnel and 
their dependents. 

Team members found that the center is 
physically, psychologically, procedurally, 
economically and cL'tjrally, accessible to 
catchment area residents. On the other hand, it 
was the team's general impressici that the center 
is not physico.'ly accessible to all residents in the 
catchment area. Even though oome services are 
provided at least 1 day a week in each county, 
there are problems with transportation and the 
services seemed to be limited in scope. 

RECOMMENDATIONS 

The site visitation team made the following 
recommendations: 

• A coordinator for consultation and education 
should be employed. 

• The center should secure a coordinator to be 
based in the regional hospital in Savannah to 
supervise admissions from the catchment 
area. A consortium of pertinent Fort Stewart 
personnel and center staff should be formed 
to develop plans of action in order to meet 
the mental health needs of Liberty County 
residents. 

• The center should secure the services of a 
planner and grants officer who could do the 
forward planning to meet the anticipated 
needs of areas such as Camden County where 
the Poseidon submarine terminal is now 
located. 

• The center should employ a business manager. 

• The alcohol and drug abuse facility should be 
operated on a 24-hour-a-day basis. 

• A coordinator for the Hinesvillo operation 
should be appointed. 

• Staff vacancies should be filled as soon as 
possible, particularly that of the coordinator 
of the alcohol and drug abuse center. 

• The center should explore methods of 
increasing awareness and visibility of its 
services in the outlying counties through a 
vigorous, ongoing public information program. 
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• Consideration should be given to expanding 
the cent€*r"s tranvortation program, 
especially in the outlying catchment area. 

STEPS TO ENSURE IMPLEMENTATION 

After the study was completed, a formal 
written report was submitted to the CACMHC. 
This report was then forwarded to the State 
mental health office for the purpose of gaining 
additional support for funding of needed staff 
positions and programs. A copy was also provided 
to the State Mental Health and Mental Retarda- 
tion Advisory Council. 

EXTENT OF IMPLEMENTATION 

To date, all but two of the recommendations 
have been implemented. A coordinator to be 
based at the regional hospital in Savannah has not 
been designated, and the tran^rtation problems 
of persons in the outlying catchment area have 
not been resolved; however, some attempts have 
been made to ease these difficukies. 

SPECIAL BARRIERS OR SUPPORTS 

This study was greatly aided by the guidance 



and cooperation offered by the CACMHC staff. 
Such assistance included training inexperienced 
team members on the application of national 
requirements and standards in mental health 
centers. The major barriers were the team 
members* limited time and the large size of the 
catchment area. Since team members Ived in 
different communities, it was difficult for all 
members to meet on a regular basis to implement 
the study. 

RESOURCES AND COST S 

Each of the 17. team nrtembers donated ap- 
proximately 25 hours to the evaluation. The 
CACMHC provided clerical assistance and 
necessary supplies to conduct the research and to 
print the final report (the estimated cost was 
$1,000). 

ADDITIONAL COf^ENTS 

All of the individuals involved in this evaluation 
felt that the project was worthwhile, especially in 
view of the successful implementation of the 
recommendations. Another evaluation was 
completed in February 1983. 



25. A Children's Committee Evaluates an Emergency 

Shelter Program 

Joan Wooiiey and Dorothy Kurjan 

Joint Children's Committee, Western Massachusetts 



SUMMARY 

Contracts for several programs funded by the 
Massachusetts Department of Mental Health 
(DMH) to provide services for children in a 
Massachusetts county were up for renewal in the 
spring of 1982.* In order to make a sound 
recommendation on the disposition of these 
contracts for the department of mental health, 
the children's committee of a Massachusetts area 
mental health board decided to evaluate the 
programs by conducting site visits and inter- 
viewing staff and other appropriate persons. One 
program in particular. The Shelter, presented 
special concerns to the evaluators. 



For furthtr biformation write Joan M. WooUey, M.S.W.. 
UCSW, c/o DMH District Office. Northampton State 
Hospital, Northampton, MA 01060, (413) 584-1644. 



*6ecause of the circumstances surrounding the outcome 
of this case, the specific location of the program eval- 
uation is not identified in order to ensure confidenti- 
ality of the program. 
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TYPE OF ORGANIZATION 

The establishment of a children's connnriittee in 
each DMH catchment area is mandated by State 
law to review ongoing services for children funded 
by the department, to assess needs for new 
programs and to advise the department of its 
findings* At the time of the evaJuations, the 
children's committee was composed of 1 1 
members — four on the area mental health board, 
and seven others (area mental health boat J 
members are appointed by the Governor). Each 
member participating in the evaluations had an 
interest in, and commitment to, children. 

EVALUATORS OR MONITORS 

In the spring of 1982, the children's committee 
consisted of providers of children's educational 
and social services, board members of various 
human service agencies, a probation officer, and a 
city councilwoman. Three members of the 
children's committee were directly involved in 
the evaluation of The l}helter: the chairperson, a 
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social worker in private practice and former 
director of a residential facility for moderately 
disturbed adolescent women; a member of the 
board of an agency serving adolescents; and a 
retiree who was formerly employed as a child 
advocate by the Massachusetts office for children 
and who now serves on boards of several human 
services agencies. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The purpose of the children's committee is to 
make recommendations on existing children's 
programs to the area mental health board, which 
then advises the department on children's issues. 
Problems at The Shelter came to the attention of 
the committee because of a deep concern among 
child-serving professionals (some of whom were 
members of the children's committee) about the 
quality of the program and the stability of the 
agency. 

LEVEL OF PARTICIPATION 

The members of the children's committee 
participated in the following activities: develop- 
ment of a uniform method for investigating all 
the programs being evaluated; formation of two 
separate four-member teams for each program 
evaluation (each team conducted one or two site 
visits); and preparc-tion of written reports that 
were submitted to the children's committee and 
the area board. In the case of The Shelter, the 
evaluating team recommended that a new 
contractor be selected. This action required the 
children's committee to meet with the three 
agencies that submitted proposals to take over 
the adminii nation of the program. A report 
describing the rationale for changing the pro- 
gram's contractor was written by the evaluation 
team and presented to the area board. 

TARGET OF EVALUATION 
OR MONITORING 

The target of the evaluation. The Shelter, is an 
emergency shelter for adolescents in crisis 
situations who need a temporary place to stay and 
an atmosphere of caring. The other objectives of 
the program are to assess the problems of these 
young people and their families, to offer remedial 
assistance, and to work with other community 
agencies in making placement decisions con- 
cerning the adolescents. At the time of the study, 
a regionvwde agency had a contract v;ith the 
department of mental health to provide youth 
services of these kinds* The Shelter, located in a 
Massachusetts city with a population of 
approximately 50,000, is the only emergency 
shelter for adolescents to serve a largely rural 
county containing a population of 143,000 and an 
area of 947.14 square miles. 
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PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The evaluation process concentrated mainly on 
the quality of services delivered to adolescent 
clients by one specific agency— The Shelter. 
After determining the goals of the program, the 
primary focus of the evaluation was on 
programing for the client, that is, from intake 
procedures to case disposition and followup. 
Administration, staff, physical plant, and use of 
community resources were analyzed in terms of 
how these components assisted in the 
implementation of stated program goals. Clients' 
rights as well as tf'e rights of biological or foster 
parents, where i^pf !icable, were also deemed 
important in the evaluation. Finally, problems 
with staff turnover and low morale were major 
concerns that observers feit could be attributed 
to the contracting vendor. 

TECHNIQUES USED 

The children's committee developed its own 
evaluation and monitoring procedures. The 
program director of The Shelter was asked to 
submit a statement of the program goals, 
descriptions of the client population, and staffing 
patterns, and the director's perceptions of the 
program's strengths and weaknesses. Site visits 
were made by the evaluating team and included 
interviews with the director and several staff 
persons and observations of clients in the 
program. When the evaluation report was 
completed, it was sent to the program director 
and an "exit interview" was scheduled with the 
director to discuss the findings and to obtain 
feedback from the program staff about the 
evaluation. 

This was the first time that the children's 
committee had completed a formal evaluation of 
DMH-funded programs, and the technique proved 
useful. Although the committee has made some 
revisions 1 . the questionnaire, the basic format of 
the instrument will not change. 

FINDINGS OF EVALUATION OR MONITORING 

At the time of the evaluation, the program 
director of The Shelter had been in that position 
for approximately 6 months. He used this period 
of time *^trengthen the administration of the 
program and to regain credibility in the com- 
munity. The contracting vendor, however, was not 
very helpful in these efforts. For example, the 
evaluating team found that there was inadequate 
backup staff available to the program, that is, 
regular line staff had difficulty taking sick leave, 
vacations, and time off for training. Staff backup 
and support were often missing during times of 
crisis, which are frequent when dealing with 
troubled adolescent populations. In general, the 
vendor was not providing adequate clinical and 
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administrative supervision and support to the 
program staff- 

In addition, the evaluating team found that the 
family coordinator position could be made more 
effective by changing his or her available hours to 
coincide with the out -of -school time of The 
Shelter's residents, The team also found that 
clinical supervision for the program's family 
counselor was lacking. 

Other team findings focused on funding issues. 
For example, all of the staff were greatly 
underpaid, which was primarily a result of 
inadequately funded contracts. On the other hand, 
the evaluating team had questions about the 
amount of administrative costs of the program in 
relation to the actual services provided by the 
vendor. The multifunding sources of the program, 
including the State department of social services 
and the department of mental health. United Way, 
the city and county, and a private agency also 
presented many problems, with little coordination 
among the six funding agencies, staff energy and 
time were siphoned off from program activities 
into time spent on additional bureaucratic 
functions. 

RECOMMENDATIONS 

The team's major recommendation was not to 
renew the current vendor's contract to provide an 
emergency shelter. As suggested by the team, the 
department of mental health should contract with 
a vendor who could provide the following; a 
staffing pattern that would include an efficient 
division of responsibilities, strong clinical backup 
for program staff, clinical staff backup and 
support at times of crisis, well-developed staff 
training programs, and strong supervision and 
backup from the vendor. Further, the evaluating 
team strongly recommended that a system of 
communications and joint monitoring and 
evaluation be established among the funding 
agencies. 

STEPS TO ENSURE IMPLEMENTATION 

The committee has considerable authority to 
implement its recommendations since the 
department of mental health reviews those 
recommendations before determining which 
programs will continue to be funded. In addition, 
the entire children's committee mil meet with 
the new vendor from time to time, and the 
evaluating team will visit the program periodi- 
cally duiing the year. 

EXTENT OF IMPLEMENTATION 

The team's primary recommendation that the 
contract with the current vendor be terminated 
was implemented. Subsequently, a new vendor 
who met the evaluation criteria of the children*s 
committee was awarded the contract to admini- 
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ster the emergency shelter. 

Another recommendat'on concerning multi- 
funding sources was also implemented. When the 
contract was transferred to a rew vendor, a 
meeting was held involving all six funding sources, 
the outgoing and incoming vendors, and members 
of the children's coinmiltee. The purposes of this 
meeting were to ensure a smooth transfer from 
the old to the new vendor, to make sure that all 
concerned parties had a clear understanding of 
the goals o7 the emergency shelter, and to 
encourage cooperation among all parties. 

SPECIAL BARRIERS OR SUPPORTS 

The emergency shelter staff responded 
positively to the evaluating team. In addition, the 
area mental health advisory board was most 
supportive of the evaluation, as well as the area 
DMH director who welcomed the evaluation. 

It was important that each team evaluating 
childrens* or ad' descent programs, including The 
Shelter, was made up of professionals from areas 
such as special education, social services, city 
government, the court system—to name a 
few— plu? concerned, aware citizens. Since the 
children's committee is a volunteer group, 
securing adequate time from the citizens and 
coordinating team schedules was difficult, but not 
impossible, to accomplish. 

RESOURCES AND COSTS 

The evaluating teams spent approximately 3 
months on the program reviews including 
preparation, site visits, team discussions, writeups 
and concluding interviews. An average of 25-35 
hours per individual was spent on each program 
review including The Shelter. The entire chil- 
dren's committee met once a week during this 
' imd, and the teams met as needed. Other direct 
costs, such as typing and reproducing efforts, 
were absorbed by area department of mental 
health staff. 

ADDITIONAL COMMENTS 

The children's committee found that a program 
needs to undergo a rigorous evaluation every 3 
years, with foUowup site visits and interviews 
with all committee members. In this way, the 
area t>oard \Adll become aware of potential 
problems and how to resolve them before the 
program is adversely affected. 

Committee members also feel that it is 
important to have an evaluation specialist 
conduct several training sessions on interviewing 
techniques. In addition, the committee lecom- 
mends that a mix of professional and concerned 
lay citizens always be used when forming 
evaluation teams. Further, the children's 
committee believes it is important for any 
evaluation team to stress the positive aspects of a 
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program and to determine how community 
advisory committees can most effectively assist 
programs toward goal achievement and provide 
ongoing support. 
The evaluation efforts of the children's 



committee made both the committee and the area 
mental healtn board more credible and viable 
within the county. It also made the DMH-funded 
providers more vigilant regarding the quality of 
the services they deliver to children. 



26. Evaluation of GCMHC Adult Outpatient Services' 
Group Orientation Procedure 

Jan Fbutz 

Citizens' Advisory ComeU, Granite Community Mental Health Center 

Barbara K. Goza 
Salt Lake County Division of Mental Health 



SUMMARY 

The citizens' advisory council and Granite 
Community Mental Health Center (GCMHC) staff 
and management collaborated in the evaluation of 
one program component vwthin the center. The 
question of interest was the cost and effective- 
ness of a newly initiated group format for 
orientation of clients to adult outpatient services. 

Three different methods of analysis were used: 
analysis of complaint files, retrospective review 
of client flow through the system, and client 
satisfaction mth the orientation. Generally, the 
group orientation procedure was found to be cost 
effective; however, there were some problems 
that needed to be addressed. Implemented 
recommendations for change included: job 
redesign for the screeners to facilitate identi- 
fication of clients needing more immediate help 
and preparing others ft the group experience, 
and additional orientation meetings and child care 
services to improve the referral system. 

TYPE OF QRGANIZATIQN 

The evaluation was conducted by the citizens' 
advisory council for the community n.^^ntal health 
center. In collaboration with center staff and 
management. The 20-person council was com- 
posed of Interested community members, 
appointed with the approval of the county 
commission which serves as the governing board 
of the center* The advisory council considers 
Itself an advocacy group—both for the center and 
for the community It serves. Membership in the 
council is cdtermlned by professional skills, 
special interest in mental health (such as 
consumers or families of consumers), or general 
interest in mental health issues. 



For further information write Jan Foutz, Primary 
Children's Hospital, 320 Twelfth Avenue, Salt Lake 
City. UT 84103, (801) 521-1221. 



EVALUATORS OR MONITORS 

Planning and implementation of this project 
was accomplished primarily by advisory council 
members, with the support of the center's 
evaluation staff. The chairperson of the task 
force had primary re^nsibility for the project. 
This included focusing the research questions, 
interpreting data, recommending changes on the 
basis of the data, and facilitating the innplemen- 
tation of recommended changes. At that time, the 
chairperson of the task force had no formal 
training in evaluation research. Other advisory 
council members who assisted in planning and 
data interpretation were two psychologists 
employed by other human service organizations, 
and a person who worked in the billing office of 
the center and *vho volunteered to perform the 
telephone survey in addition to her regularly 
scheduled work requirements. This person was 
taking a course in research design at the time and 
was able to use this hands-on survey experience 
to fulfill class requirements. Center evaluation 
staff assisted with the design and data analysis. 

REASONS FOR THE EVALUATION 
OR MCNITORING 

GCMHC had tried an individual approach to 
orientation of clients to center services and had 
been dissatisfied with the failure of this program 
to reduce the frequency of broken appointments. 
The adult and family outpatient unit started a 
new orientation process to try to maximize staff 
involvement with clients and to maintain 
appropriate services to clients. This new program 
used a group format. One year after the program 
had been implemented, there were some client 
complaints from a person (or persons) who felt 
s/he was asked to attend a group orientation when 
s/he was in crisis and should have received 
clinical attention more quickly* In keeping with 
the philosophy of the center, the executive 
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directcr informed the advisory council of the 
complaint. The unit manager asked an advisory 
council member to chair a committee to plan 
research to examine the extent of the problem 
and potential solutions. 

LEVEL OF PARTICIPATION 

The advisory council was very much involved in 
planning the three-pronged approach to the 
research. Council members selected the research 
question, reviewed proposed research designs, and 
established project timelines. Center evaluation 
staff analyzed complaint Tiles and data on client 
flow through the intake system. A student 
volunteered to conduct a telephone survey of 
consumers. The advisory council was responsible 
for compiling tliese three sources of data, 
developing a trusting relationship with the team 
leader responsible for this program component, 
interpreting the data, and making recommenda- 
tions for change. 

TARGET OF EVALUATION OR MONITORING 

The target of this research was the adult and 
family outpatient services unit in Granite 
Community Mental Health Center in Salt Lake 
City. Utah. The agency is located in an urban 
community and treats clients on an outpatient 
basis with individual and group psychotherapy. A 
broad range of client problems are treated, 
including family conflict, depression, alcohol and 
drug dependency, and more chronic conditions 
typically encountered in urban settings. The 
specific as^t of the evaluated unit was a newly 
initiated group format for client orientation. The 
group format had been planned in an effort to 
provide a more cost-effective procedure for 
adequately disseminating general information 
about GCMHC services and alternatives. It was 
hoped that the orientation would reduce the 
number of scheduled intake appointments not 
kept. Center management and the advisory 
council were interested in determining whether 
thece objectives were being met. particularly for 
low income and minority clients. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The general issue evaluated In this study was 
the effectiveness and appropriateness of a newly 
initiated group format for orienting clients to 
center services. First, the number of client 
complaints concerning this new format was 
compared with the number of complaints arising 
from the previous format that utilized individual 
orientation sessions. Second, appointment 
foUowthrough rates for clients attending indivi- 
dual and group sessions were compared. Third, 
client satisfaction with the new format was 
examined. 



TECHNIQUES USED 

Thr'^e separate techniqu 3S were used in this 
study. Two of these techniques involved analyzing 
data that were already available at the center. 
One involved the coliectic-n additional data. 
Data on client complaincs were obtained from an 
ongoing file of complaints maintained by ths 
center director. The number of comolaints 
received during each of the 6 years prior to, and 1 
year following, initiation of the group orientation 
program was compared. Appointment follow- 
through rates were calculated by analyzing data 
from the center's client information system. A 
member of the center's research staff reviewed 
data on client flow from initial call to first 
treatment session. All clients scheduled for 
individual intake sessions and for group orienta- 
tion meetings during the previous 1-year period 
were compared with regard to appointment 
follow^;hrough. Finally, client satisfaction data 
were collected by means of a telephone survey. A 
member of the center's billing staff volunteered 
her time to conduct the telephone survey. An 
attempt was made to interview all 37 clients 
scheduled for group orientation sessions during a 
2-month period. Interviews were conducted 2 or 3 
months following the scheduled orientation 
sessions. 

FINDINGS OF EVALUATION 
OR MONITORING 

Several findings emerged as a result of this 
three-pronged evaluation. The first was that the 
number of client complaints had not increased 
frllowing the introduction of the group orienta- 
tion format. Secondly, the available data from the 
client information system indicated that the 
proportion of clients keeping thfeir first treatment 
appointments following group orientations was at 
least equal to, if not greater than, the proportion 
of clients doing so after individucl intake sessions. 
In addition, the client tracking data suggested 
that male clients were more likely to attend their 
scheduled orientation sessions than were female 
clients. 

Findings from the client satisfaction survey 
presented a somewhat more complex picture. Of 
the 37 clients scheduled for group orientation 
sessions. ?n had attended. The two major reasons 
for nonattendance were conflicts with work 
schedules and the clients' perceived needs for 
more immediate clinical attention. Clients were 
asked to rate various aspects of center services in 
general, and the orientation sessions in particular, 
using a Likert-type scale. This question format 
asks clients to respond to items using a scale with 
inter/als ranging from very negative to very 
positive. In the present case, a seven-point scale 
ranging from poor (1) to excellent (7) was used. 
When clients were asked to rate the change in 
their presenting problem since contacting the 
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center, the mean response of those who did and 
did not attend their sc -jeduled orientation sessions 
were equal and indicat3d slight improvement. 
Clients were not satisfied with the extent to 
which their questions regarding center services 
were answered. They were somewhat less 
satisfied with the extent to which their specific 
questions/concerns were addressed. A somewhat 
less than neutral response was given to the 
question regarding the atmosphere of the 
orientation. When asked what would have made 
them most comfortable, 55 percent of respon- 
dents reported a need for more privacy for 
discussing personal problems. An additional 30 
percent reported that nothing would ha>'o helped, 
that "it's just difficult to ask for this kind of 
help.** Of the 20 respondents v^o attended the 
group sessions, 1 1 were niven intake appoint- 
ments, 3 were referred to private clinicians, and 6 
reported not being given specific suggestions for 
referral. 



RECOMMENDATIONS 

The conclusions drawn from these data were 
that the group orientation procedure was cost 
effective from the perspective of staff deploy- 
ment. However, there seemed to be a number of 
changes that miyht be made in order to meet the 
needs of the clients more adequately. These 
changes v/ere: 

1. To expand the scheduled group orientations 
to alternative times (perhaps evenings) to 
alleviate work schedule/child care conflicts. 

2. In order to improve accessibility to women, 
screening staff should be sensitized to the 
needs of women attending orientation 
sessions, and child care services should be 
considered. 

3. Screeners should prepare clients extensively 
for accurate expectations of the orientation 
meeting, especially that it mil not involve 
much personal sharing. The orientation 
sessions should be kept small. More staff 
should be available so that the clients can 
break into smaller groups of four or five for 
discussions. 

4. The problem that some people do not have a 
clear idea of alternatives for help following 
the orientation session should be addressed 
by: providing more individual attention and 
support in looking at alternatives; having 
intake hours available right after the 
orientation session so that those who do not 
want to wait c< n go right into intake; and by 
improving the referral system through 
improved communication between the agency 
and private practice clinicians. 
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5. The issues of payment and fees may be of 
such a different quality than issues of 
treatment and services, that it might be good 
to have the two orientation staff members 
divide discussion of these issues oetween 
them. 

6. While the data from these studies do not 
dir^'^tly address problems of low income or 
minority clients, the client flow data, 
indicating greater followthrough by men may 
be an indication that the orientation 
meetings are more effective for individuals 
socialized into the white male culture. This 
points to the importance of continued 
sensitivity to cultural issues in the screening 
process. 

STEPS TO ENSURE IMPLEMENT AlION 

The steps considered to be most important to 
ensure implementation were: early involvement of 
the staff and management of the unit being 
evaluated; development of communication and 
trust between the advisory council members and 
the first level of supervision for the program 
component; and feedback and discussion of data 
generated in the evaluation by the unit staff and 
management. Since the recommended changes 
were under the control of unit management and 
staff and the suggested changes were implemen- 
ted, it was not considered necessary to publish 
these data outside the center. 

EXTENT OF IMPLEMENTATION 

Many of the recommendations from this 
research process were acted upon by unit staff 
and management. The unit assigned two persons 
to share the screening responsibilies so that they 
would also have the opportunity to participate in 
other forms of therapy, thereby avoiding reduc- 
tion of their work routine. This job redesign was 
considered helpful in keeping screeners* attention 
focused on the special needs of clients. A child 
care service, staffv?.d by volunteers, was begun by 
the volunteer coordinator an j was available 
during the hours of the group orientation. A series 
of open house meetings between private practice 
clinicians and center staff, management, and the 
advisory council, was organized and run by the 
advisory council. The council chairperson of this 
research felt that the recommendations were 
implemented as much as possible by the center. 

SPECIAL BARRIERS OR SUPPORTS 

Several factors supported this evaluation. The 
responsive management that initiated the 
research indicated a readiness to use evaluation 
data. It was very important to have represen- 
tatives of the advisory council, the service 
delivery system, and research and evaluation staff 
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involved in the planning and disseminating phases 
of the research. Per'.aps most important, the 
advisory council has established trusting rela- 
tionships with management which greatly 
facilitated their collaborative work. Trust 
building takes time, but since evaluation can be 
threatening, it is a critical step to take before 
any negative feedback can be given in a con- 
structive way. It was also valuable in this 
research effort that the chair of the advisory 
council committee was a lay person inexperenced 
in research. Her questions about how the research 
design would answer the questions were of real 
value. Another characteristic of this corr.mittee 
which contributed to implementation of the 
recommendations was that the members were in 
no way "yes" people. They were able to be verbal, 
interest^, and supportive, but supportive in a 
challenging way. 

The only barrier to the work was the difficulty 
in understanding the client flow data, given the 
relatively large number of persons "lost" from the 
system. However, this barrier became the 
stimulus .*or further research and for improving 
the client tracking system in the center. 

RESOURCES AND COSTS 

The amount of time contribu' ed to this 



research was approximately: 250 hours for 
advisory council members in planning and 
implementing the evaluation; 160 hou-s for 
evaluation staff in reviewing complaint and client 
tracking data and providing consultation for the 
research: and about 100 hours from the student 
volunteer who collected the data for the tele- 
phone survey. 

ADDITIONAL COMMENTS 

In evaluating her involvement in this research, 
the chairpf^rscn of the task force thought that the 
research had been effective in improving service 
delivery. She had attended the group orientation 
both before and after the implementation of 
recommended changes and thought the process 
was much improved. Immediately after the 
screener job had been redesigned so that two 
persons were sharing these responsibilities, more 
clients began showing up for the group orientation 
and satisfaction with therapy increased. The 
chairperson also found the process helpiul to 
herself, in that it taught her about management's 
goals and responsiveness as well as the basics of 
research. 



27. A Descriptive Evaluation of Psychiatric Emergency 
Services in Two Hospitals: A Preliminary Study 

Elizabeth Fulton and Bruce Hirsch 

Mental Health Association of San Francisco 



SUMMARY 

This study was conducted by volunteers from 
the Mental Health Association of San F ancisco 
(MHASF). The purpose of the study was co obtain 
a descriptive evaluation of psychiatric emergency 
services (PES) facilities through site visits to two 
San Francisco hospitals. The study focused on the 
ways in which the physical plant, staffing patterns 
and procedures at each hospital affected the PES 
mission of providing mental health services for 
persons in a psychiatric crisis. 

TYPE OF ORGANIZATION 

The Mental Health Association of San Francisco 
is a nongovernmental, nonprofit volunteer 
organization affiliated with the Mental Health 
Association of California and the National Mental 



For further information ynite Sharon George, 317 
Morning Sun Avenue, Mill Valley, CA 94115, (415) 
381-2953. 
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Health Association. MHASF advocates for 
improved mental health services and seeks to 
educate the public about mental health and 
mental illness. The association is governed by a 
board of directors that is chosen from its 800 
members who represent the cultural diversity of 
the San Francisco community. 

EVALUATORS OR MONITORS 

This study was carried out by MHASF's public 
affairs committee in cooperation with the 
directors of the hospital PES units being studied. 
The committee organized a group of eight 
volunteers to implement the evaluation including 
an attorney, three counselors, a mental health 
administrator, an insurance professional, a 
housewife, and an organizational consultant. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The study was initiated as a result of MHASF 
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discussions concerning long-range strategies for 
reviewing co:.imunity mental health services 
programs in San Francisco. Members of the 
MHASF public affairs committee and board of 
directors recognized that psychiatric emergency 
services (PES) facilities serve as the hub for 
community mental health services since they 
provide triage and access to other programs. As 
such, it was vital to evaluate certain aspects of 
PES facilities. 

LEVEL OF PARTICIPATION 

Volunteers prepared and submitted a formal 
proposal to the San Francisco Department of 
Public Health requesting permission to conduct 
the study. They also conducted a lengthy review 
of regulations governing the operation of the PES 
facilities. The eight members of the visitation 
task force worked with the public affairs 
committee in designing the data collection 
instrument and in issuing a final report; they also 
worked with the directors of the two PES units 
and their staffs in implementing the evaluation. 

TARGET OF EVALUATION 
OR MONITORING 

The study covered two PES units that are part 
of the community mental health services networK 
serving the city and county of San Francisco: the 
San Francisco General Hospital PES and the 
Westside Crisis Clinic at Mt. Zion Hospital. San 
Francisco General Hospital is located in an 
ethnically diverse, low income area: it covers 
more than one catchment area and handles the 
overflow of clients from other districts. The 
predominant population groups using this crisis 
unit are v^^ile, black, Hispanic, and Chinese. Mt. 
Zion hospital is located in a predominantly low 
income area although its catchment area also 
includes some high-income sections. Major client 
groups using the PES unit at Mt. Zion are black, 
white, Chinese and Filipino. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Psychiatric emergency services serve as the 
entry point of the mental health treatment 
system for people experiencing a psychiatric 
crisis. On the basis of site visits, this study sought 
to evaluate the following: the ability of PES units 
to meet minimum standards for pleasantness of 
physical surroundings such as lighting, decor, and 
availability of reading materials; ability of these 
units to cope witn a sudden influx of patients: 
actual client contact; stressfulness of intake 
procedures; noise level, ventilation, and temper- 
ature: condition of seclusion rooms and re- 
straining equipment; staffing patterns, including 
seXt ethnicity, and linguistic capabilities; and 
average time spent with clients. 
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TECHNIQUES USED 

Volunteers first met with each PES facility 
administrator in nrdsr to become familiar with 
the operation of the units. Next, they reviewed 
regulatiors governing the operation of the 
facilitie? and developed a checklist to be used to 
evaluate the units during site visits. Checklist 
items included dates and times cf patient visits, 
available physical equipmunt, staff structure, 
availability of staff, and patient information. 
Patient information included date of entry, 
ethnicity, age, sex, and entering condition as 
observed by volunteers. Site vi^'ts were scheduled 
during known periods of stress when people would 
be inclined to seek out the services of a service 
unit. Finally, summaries of the visits were 
produced from data recorded on the checklists. 

FINDINGS OF EVALUATION 
OR MONITORING 

The study produced the following findings: (1) it 
strengthened MHASF recognition of the central 
role of psychiatric emergency services (PES) in 
the public mental health delivery system; (2) it 
highlighted the importance of using non- 
English -speaking clinicians in a culturally diverse 
city like San Francisco; (3) it increased the 
mental health association's respect for and 
understanding of PES staff; and (4) it highlighted 
the severe space limitations at one unit that made 
it difficult to handle an influx of clients. 

RECOMMENDATIONS 

The results of the general site visitation 
suggested that there was a need for a followup 
study to identify the availability and appro- 
priateness of referrals for persons entering the 
mental health system through tha PES units. Also, 
it was reccmmerded J-hat the physical space 
problems at one unit be addressed. This recom- 
mendation was made to the director of the unit 
and was supported by city officials, including the 
program chief of community mental health 
services and the director of the department of 
public health. 



STEPS TO ENSURE IMPLEMENTATION 

In order to ensure that the followup study would 
take place, MHASF and the directors of the crisis 
units worked together In planning the new study. 
The association continues to evaluate and monitor 
other community mental health services programs 
and to support efforts to provide more physical 
space for crisis units. MHASF representatives 
have observer status on the community advisory 
board for San Francisco General Hospital and 
have been given a regular place on the mc^thly 
ayenda. 



EXTENT OF IMPLEMENTATION 



RESOURCES AND COSTS 



The followup study was conducted and is 
described in Chapter Six of the Casebook. 
Moreover, the new and expanded PES unit opened 
in 3ulyt 1982. 

SPECIAL BARRIERS OR SUPPORTS 

Some volunteers were unfamiliar with the 
operations of psychiatric emergency services 
(PES) facilities and the mental health treatment 
system. On the other hand, hospital officials were 
very helpful and seemed to welcome the objective 
of the volunteers. The director of one PES has 
established a strong r ngoing relationship with the 
Mental Health Association of San F rancisco 
(MHASF). 



Planning the study took several months, and 
site visits were conducted by eight volunteers 
over a period of 4 days. Visits were arranged to 
cover a representative sample of all times of the 
day. Volunteers were supported by MHASF staff 
as part of its ongoing public affairs program. 

ADDITIONAL COMMENTS 

The study showed that data collection by 
volunteers can be important and useful to the 
professional community. One inipnrfant lesson 
learned from this experience is that a single 
program may require more than one evaluation 
effort. 
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What Will Be Your Focus? 



The decision regarding the scope of the evaluation has now been 
made, and Doug Brown and his committee must now decide the issue of 
topics they will address during the course of the study. They know that 
they are concerned with service delivery across county programs, but they 
need to develop a perspective from which to view the system. It will be 
from this perspective that research questions can be developed and a sub- 
sequent research strategy can be designed. One of the committee members 
mentions that any evaluation activity aimed at long-range planning will 
have to take into consideration the State's plans for the future of River- 
view State Hospital. On further investigation, Doug confirms that State 
officials are planning to close a vong of Riverview State Hospital during 
the next fiscal year. This is particularly significant for Greene County 
since the unit slated for closing houses a majority of county residents. Al- 
though services in the community have expanded and improved during the 
past several years, it is not clear whether the county program can rapidly 
absorb more seriously disabled, long-term institutional residents. It seems 
clear to Doug and tlie corrunittee that their analysis should focus on the 
issue of deinstitutionalization and that the research should be designed 
with this topic in mind. 

Having selected a topic or issue to explore, Doug and the committee 
members must now decide what aspects of the issue are of particular con- 
cern to the mental health system in Greene County. By going through this 
process they can isolate the questions to be answered by the evaluation and 
can therefore determine what steps they will take to get the answers. 
After conversations among themselves and staff at the county level, they 
decide to explore the following problem areas: 

To what extent are services available in Greene County to meet 
the needs of current residents at Riverview State Hojpital? 

What new services need to be developed to meet these needs? 

To what extent will coimty residents resist the placement of formerly 
institutionalized persons back into the community? 

/ administrative procedures established between county program 
staff and institutional staff tf ease the transiton? 

Has an adequate amomt of money been allocated to follow former 
residents back into the community? 



The cases included in this chapter cover a wide 
s;>ectrum of topic areas, from funding issues to a 
concern about the proposed closing of a State 
facility. The diversity of aieas shows the breadth 
of topics and issues that can be addressed by 
citizen and consumer groups. 

Needs Assessment 

The first case, by Lois van Valkenburgh, relates 
the efforts of the Alexandria (Virginia) Com- 



munity Mental Health Center's governing board to 
assess community needs in preparation for the 
following year's program planning. In addressing 
which issues shoidd be covered by the needs as- 
sessment, board members, together with staff, 
identified the critical areas. The board focused on 
four major issues: truant youth, outreach to mi- 
norities (especially important given the recent 
influx of Indochiness refugees in the area), out- 
reach to the elderly, and supervised residences for 
the chronically mentally ill. Interviews were held 
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with key informants in the area, and the board 
worked with the center's evaluation staff to de- 
velop recommendations and prepare a report. 

Administrative Priorities 

Ed Diksa of the Sonoma County (California) 
Mental Health Board discusses that board's role in 
providing citizen participation in the county's 
mental health planning process and in approving 
the process whereby the plan is adapted. During 
the review, two major issues were investi- 
gated—budget cuts and friction between agency 
staff and the director. It is interesting to note 
that board members felt obliged to respond to 
staff morale and quality issues as part of their 
mandate to evaluate and plan for county services. 
Meetings were held with administrators and pro- 
gram directors, community services, and staff. 
Results were reported to the mental health de- 
partment and the county board of supervisors. 

Facility Concerns 

Jan Holcomb, Ann Nera i, and Karen Helfrich of 
the Mental Health Assocfc cion of Greater Chicago 
discuss the work of that organization's site visi- 
tation committee in monitoring car^ at six State 
mental health hospitals. Individual interest in 
monitoring the facilities, as well as continuing 
media coverage of the inadequacies of the State 
hospitals, prompted association volunteers to fo- 
cus their review on specific aspects of the fa- 
cilities. Site visits cover a wide range of issues, 
from policies and procedures to housekeeping, 
staff attitudes, and residents' rights. Trained 
volunteers directly observe and interview patients 
and conduct interviews with staff and adminis- 
trators. They report findings and recommen- 
dations to the facility superintenCont, share in- 
formation with the Illinois Guardianship and 
Advocacy Commission, and, in one case, the press. 

Funding and Resource Allocation 

Two cases address the issue of how funding de- 
cisions are made. Jean Abruzzlrio and John Cor- 
rigan report on the sophisticated computer deci- 
sionmaking tool used by the Franklin County 
(Ohio) Mental Health Board to set service prior- 
ities and recommend allocations. The board, man- 
dated by State law, is composed of area citizens 
who plan, recommend funding levels, evaluate, 
and oversee the development of services in the 
county that are delivered by contxact agencies. 
Resource allocation decisions are only one aspect 
of the board's evaluation responsibilities, but in a 
complex and diverse service system, such as 
Franklin County, resource issues are of paramount 
concern and take an inordinate amount of board 
members' time. The case stimmary describes the 
use of a computer tool, the goal programming 
model, which provides greater flexibility to board 



members in their decisionmaking by analyzing the 
impact of various funding strategies and providing 
information on service availability, need, fiscal 
issues, and special constraints. 

The second case concerning funding comes from 
Frances P. Meehan of the liaison committee of 
the San Gabriel Valley (California) Mental Health 
Association. As in Franklin County, it stres^^^ss the 
importance of understanding resource allocation 
issues. When the committee discovered a disparity 
between allocation of mental health funds to their 
area as compared to other regions in the county, 
ti:ey reviewed pertinent sections of the county 
budget and compared this to the size of the pop- 
ulation and projected needs. A report on these 
issues was made to the county supervisor repre- 
senting San Gabriel Valley, plus a nimiber of civic 
and advocacy groups, since prior appeals to the 
county mental health department had been 
unsuccessful. 

Service Delivery 

The next case study, presented by Barbara Todd 
of the Fairfax-Falls Chur< h (Virginia) Community 
Services Board, covers the work of a task force to 
explore whether systemwide consolidation of 
services could reduce costs sufficiently to main- 
tain existing service levels in a time of reduced 
budgets. The task force conducted site visits, re- 
viewed written materials, talked to staff, and 
generally undertook a very ambitious project. 
Perhaps the project was too ambitious, since very 
few of the task force recommendations were im- 
plemented. Results ipc^vded an overview of how 
well clients wer**- being served and the feasibility 
of systemic change. 



Individual Rights 

The Montana Mental Disabilities Board of Vis- 
itors reviews annually the State's community 
mental health centers, inpatient facilities for the 
mentally ill, and institutions for the develop- 
mentally disabled. Kelly Moorse writes tliat the 
board examines patient care and treatment, med- 
ication, recordkeeping, and consumer issues. 
Members conduct site visits at facilities, review 
treatment and medication plans, inspect resi- 
dential 'nd treatment areas, interview staff and 
patients, and respond to individual grievances 
from patients, family members, and the legal 
system. Ensuring that patients' rights are pro- 
tected is of primary concern to the board. The 
loard is an imporU it vehicle for patients vishing 
to resolve individual complaints, and it also ad- 
di-esses client-right issues on a ttatewide level. 
Site-level findings are sent to the facility staff 
for comment prior to the preparation of a final 
report; however, individual grievances are eval- 
uated immediately and options for change pre- 
sented to the client and others involved. 
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Client Satisfaction 

Joyce G. Smith of Hill House Mental Health 
Rehabilitation and Research, Inc., discusses a 
project whereby clients of a psychosocial reha- 
bilitation agency designed instruments to assess 
client needs arid evaluate client progress. The 
Client-Oriented Program Evaluation (COPE) 
group consists of current or former clients who 
provide evaluative intormation to both the staff 
and the board of trustees. In this example of 
consumer satisfaction, clients worked coopera- 
tively with the agency's research staff to develop 
better measures of client needs and program 
evaluation. Clients, were not passive reactors to 
the evaluation but instead participated in many 
complex tasks, including designixig and testing 
survey instruments. 

Special Populations 

Two cases focus on evaluations of services to 
specific recipient groups, the chronically mentally 
ill and the aging. Cheryl Fanning of the Com- 
munity Support Project (CSP), Arizona Depart- 
ment of Health, discusses the review by three 
stdvocacy groups of a community residential 
treatment system for chronically mentally ill in- 
dividuals. A review of programs serving the 
chronic population was tied to the CSP staff's 
planning and budgeting responsibilities. The re- 
view enabled consumers to reflect on the quality 
of services provided to this target group. Re- 
viewers conducted site visits to programs in seven 
cities, focusing on physical aspects of the resi- 
dences and activities available, and they observed 
strT interactions with residents. Results were 
reported to the CSP and, in turn, to contract 
agencies. 

The second case is reported by Margaret Mim- 
ford of the In-Home Services Committee of the 
Washington Comty (Oregon) Coimcil on Aging. 
She writes about the committee's evaluation of a 
mental health clinic's ability to serve elderly 
residents of nursing homes. The committee had 
jurisdiction because of its advisory role to the 
council on aging which, in turn, contracts vdth the 
clinic. In this particular case, the focus of an or- 
ganization representing senior citizens in a 
nursing home program is self-evident. What js 
somewhat unusual is the use of a member of the 
senior citizens organization to evaluate si'ich 
concerns as the quality of life of nursing home 
residents. The committee distributed a ques- 
tionnaire to clinic staff, held consultations with 
nursing home supervisors, and attended a group 
therapy class. Findings the evaluation were 
reported to the council on aging as it considered 
whether to renew its contract with the clinic. 

Service Quality 

Ruth M. Taylor of the Parent Monitoring Com- 



mittee, Association for the Macomb-Oakland 
(Michigan) Regional Center, reports that the 
committee, composed of persons with relatives 
living in community group homes, has monitored 
over 90 group homes to date. Service quality is- 
sues are of foremost concern to this committee 
si ^e it serves as the eyes and ears for all parents 
and for the service system as a whole* The com- 
mittee's monitoring efforts also coincide with 
other quality assurance reviews implemented by 
the State. Monitoring site visits are made, an 
evaluation form filled out after the visit, and the 
report revic by a core conunittee. The com- 
mittee focuses on quality of life and environment 
issues, health, nutrition, and client rights. Rec- 
ommendations stenmiing from the site visits are 
made to the regional center, which contracts with 
the homes. 

Planning 

Each year, Marin County (California) Com- 
munity Mental Health Services establishes citizen 
task forces as part of its planning process, Ralph 
W. Accardi, of the Older Adult Services Advisory 
Committee, Marin County Mental Health Advi- 
sory Board, reports that task forces make recom- 
mendations based on needs assessment, studies of 
service delivery, and evaluation of progress made 
towards implementing the previous year's plans. 
In this case, citizen involvement in planning is 
required by tiie State. It is also evident that Marin 
County staff, togethc • with citizens, have made 
this part of a continuing process of planning and 
evaluation. 

Another planntng-oriented case is reported by 
David L. SUver and Donald A, Craig from North- 
east Kingdom Mental Health Service, Inc., in 
Vermont. Silver, who chairs the program and 
evalij.tlon conunittee of the agency's board of 
directors, and Craig, who is a staff planner and 
evaluator, describe the annual interviews with 
agency staff that are conducted by the com- 
mittee. Again, the staff/citizen collabr/ration 
surrounding agency planning appears to influence 
the setting of goals and priorities for the com- 
munity as a whole. Results from these interviews 
lead to recommendations to the board and later to 
the agency's administrative staff as they plan for 
the fuMore. 

Acceptability/ Awareness 

Kevin M. Kindelaii, Jeorge F. Mailly, and Kathy 
B. Hayes, of Winter Haven (Florida) Hospital 
Community Mental Health Center, report on a 
key informant survey that wab employed to de- 
termine perceptions of mental health needs as 
well as the agency's visibility, acceptability, and 
accessibility. The study was undertaken by the 
center's program evaluation and research com- 
mittee and, as opposed to other cases in this book* 
involved citii-ans only in the role of survey re- 
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spondents. Respondents were identified as in- 
<Uviduals in the conununity who were aware of its 
mental health problems and needs and worked in 
areas such as school counseling, social services, 
ministry, law enforcement, and homemaking. The 
use of a broad spectrum of knowledgeable indi- 
viduals in the needs assessment provided agency 
staff with baseline lata ^^th vAdch to conduct 
other targeted assessments (e.g., child-oriented 
key informants) in the future. R«?commendations 
arising from this stu<fy were iobmitted to the 
center director. 

Institutiofial Closure 

William P. Benjamin, President of the Central 
Islip Board of Visitors in New York State, reports 
on the cox:duct of an evaluation of a proposal to 
close the Central Islip Psychiatric Center. The 
board, plus an ad hoc committee of employees, 
patients, and community residents, studied the 
type and level of care being provided patients at 
the center, analyzH the possibility of patients 
receiving the same care at other facilities, and 
evaluated the effects of relocation and transfer 
on patients. Dau were gathered from key in- 
formant interviews, interviews vdth staff of that 
center and other facilities in the region, and a 
demographic study. Results were communicated 
to the State office of mental health, boards of 
other State hospitals, community and advocacy 



groups, tl.e Governor, and special legislators. 
Even though the boeu:d*s recommendation against 
closing the center was not accepted by tiie State, 
special consideration was given to the problems 
identified in the board's study (e.g., relocating 
fragile elderly patients) in the final decision re- 
garding the facility. This action was a credit to 
the board's thorough and credible evaluation of 
the proposed closing of the facility. 

Accessibility 

The work doixe by the High Point (North Caro- 
lina) Mental Health Association to assess the need 
for expanded emergency services and later to 
monitor the delivery of those services is discussed 
by Barbara Geddie. The association's focus on 
accessibility of such services came at a critical 
time. Both private and public resources, including 
the National nstitute of Mental Health, were 
aware of the deficiencies in emergency services. 
By interviewing magistrates, patients, and local 
hospital, police, and other officials, the moni- 
toring team determined that services were re- 
sponsive and effective during daytime hours, but 
tliat after office hours services could only be ob- 
tained 20 miles away. The mental health associ- 
ation board recommended an alternative plan for 
better coverage and supported the center's re- 
quest for increased funding to provide that 
coverage. 



28. Citizens Review the Needs of Special Populations 

Lois Van Vaikenburgh 

Alexandria (VA) Community Mental Health Center 



S UMMARY 

The Alexandria Community Mental Health 
Center ( ACMHC) has en 8- year reducing Federal 
grant, now part of a State block grant. The center 
is presently in its fourth year of the grant. In 
1980, before preparing the second-year grant, the 
board determined that a needs assessment was 
necessary to give direction to programs at the 
center* The results of the needs assessment were 
included in the third-year grant application and 
implementation was reported in the fourth-year 
application. 

TYPE OF ORGANIZATIO N 
The governing board of the Alexandria Com- 



For further information write Lois Van Vaikenburgh. 
Governing Board,. Alexandria ConunurJ*y Mental 
Health Center. 206 North Washington Street. 
Alexandria. VA 22314. (703) 838-4455. Home address: 
3512 Halcyon Dr.. Alexandria. VA 22305. (703) 
836-3596. 
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munity Mental Health Center is composed of 15 
community members balanced with respect to 
age, sex, economic level, and ethnic composition, 
as well as interest in mental health, mental 
retardation, and substance abuse. 

EVALUATORS OR MONITORS 

The study was conducted by the research and 
evaluation division of the center and the full 
membership of the governing board. Interviewing 
was done by the board's members: two lawyers 
(one retired), one college student, two house- 
wives, eight persons working full-time in various 
jobs (none of which were in mental health), and 
one retired civil servant. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The ACMHC governing board iu responsible for 
establishing center policy and overseeing '^entbr 
programs and finances. The governing board 
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continually monitors center programs. The needs 
assessment addressed certain areas that the 
center's board and staff identified as requiring 
further attention, 

LEVEL OF PARTICIPATION 

The board was involved in eveiy aspect of the 
study. Members first met to discuss areas to be 
addressed in the study. A number of areas were 
discarded as not requiring community assessment; 
others were thought to be too limiting. The 
director felt that residences for the chronically 
mentally ill and the problem of youthful truancy 
in the schools were not being addressed by the 
center's staff and board and that these were areas 
of greatest concern to other public and private 
agencies in Alexandria. The board settled on four 
target areas: truant youth; outreach to blacks and 
other minorities, such as the increasingly large 
populations of Asian and Hispanic backgrounds in 
the city; additional outreach to the nonpoor 
elderly population; and supervised apartments or 
residences for the chronically menially ill. The 
board assisted in designing a questionnaire. With 
the research and evaluation staff they determined 
that the interviews would be confined to "key 
informants." These would be community leaders, 
selected by board members, who dealt with the 
target populations in either a helping or moni- 
toring manner. Board members conducted 90 
percent of the interviews. Staff confined their 
interviews to colleagues within the city govern- 
ment and in helping agencies and prepared a final 
report for the board. Board and staff developed 
program recommendations together. 

TARGET OF EVALUATION 
OR MONITORING 

Alexandria is a community of about 10^,000 
persons, 22 percent of whom are black and 
approximately 3 percent Asian and Hispanic. 
Nearly 22 percent of the population is over 65 
years of age and nearly 20 percent of the families 
have incomes of less than $20,000. Other mental 
health resources in the city include private 
practitioners, one hospital with an inpatient 
psychiatric unit, three or four small private 
agencies, and the city's health department. 
ACMHC provides the following services: out- 
patient, inpatient, partial hospitalization, 2/i-hour 
emergency, diagnostic and evaluation, and 
consultation and education. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The study addressed four specific areas of 
concern to the board and staff: (1) locating and 
treating truant youth; (2) outreach to minorities 
and refugee populations through community 
leaders; (3) community residences for the 
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chronically mentally ill; and (^) consultation and 
education activities for nonpoor elderly persons. 

TECHNIQUES USED 

Based on their interests, board members 
elected to participate on one of four committees, 
each addressing one of the four areas of concern. 
A staff member was assigned to each committee. 
Other volunteers with special expertise were 
recruited by board members to assist in the 
interviewing. A single interview form was 
developed. On the advice of the board members, 
the instrument was modified slightly for each 
target group. The key informants, or community 
leaders, were selected by board members who 
were knowledgeable about the community. Key 
informants included clergy, school officials, and 
city employees. Fifty-three personal interviews 
were conducted, each lasting anywhere from X to 
3 hours in length. Orientation and training of 
board members was conducted by research and 
evaluation staff and the center director. Tools for 
the interview included copies of the center's 
annual report, the interview form, and the 
volunteer and client handbooks. 

FINDINGS OF EVALUATI ON 
OR MONITORING 

Board members felt that the interviews were 
useful in determining the key informants' 
perceptions of community needs and the level of 
satisfaction with the center. Generally speaking, 
the selected interviewees proved to be well- 
informed persons with knowledge of the needs in 
the targeted areas. 

The study indicated that there were distinct 
needs within minority groups, particularly among 
the Indochinese and Hispanic populations. 
Community leaders and social workers need to 
develop techniques to help these populations learn 
about the .<arvices evai^ ible to them. 

Statistics were supplied by key informants 
indicating the number of chronically mentally ill 
persons needing residential services. Interviews 
were conducted to determine availability of 
housing in the city, and the type of housing 
required by chroriically mentally ill individuals 
(e.g., supervised apartment, group home, etc.). 

Those concerned with elderly persons em- 
phasizea the perceived stigma of attendlny a 
mental health center and the need for additional 
outreach services into the high-rise apartment 
and condominium areas of the nonpoor elderly. 
Teachers, school administrators, staff from the 
courts and social services, as well as private 
citizens, noted the lack of day treatment for 
truant youth and the need for close cooperation 
among the schools, courts, social services, and the 
center. Identified causes for truancy were: the 
home and parents, family conflict, broken homes, 
single-parent households and the like. Training in 
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coping skills and developmental processes, drug 
counseling, and the building of self-esteem and 
self-awarencss were cited by informants as neeos* 

RECOMMENDATIONS 

As a result of the needs assessment, board and 
staff set goals for the third year of the grant in 
all four areas. Recommendations included: 

• Planning additional outreach activities for 
the elderly 

• Development of consultation and education 
programs for bilingual community workers in 
order to reach minority pc. ^lalions 

• Development of a strong cooperative 
relationshio between junior and senior high 
schools and the center's youth day treatment 
program; and expansion of consultation and 
education activities to parent/adolescent 
relationships 

• Development of supervised housing for 
chronically mentally ill with an emphasis on 
apartment living, since housing availabilit> 
within the city is extremely tight 

STEPS TO ENSURE IMPLEMENTATION 

The board, through monthly meetings, con- 
tinued to monitor the targets selected* In 
preparation of the fourth year of the grant, the 
board asked staff for reports on the success of 
their efforts in reaching the four populations. 

EXTENT OF IMPLEM£NTATIQN 

The following changes were made: 

• The director hired a transitional living 
coordinator to find apartn*ents for the 
chronically mentally ill. 



• An adolescent day treatment program was 
added to the center's partial hospitalization 
program. 

• The consultation and education staff 
continued its outreach programs at senior 
centers and added nursing homes, retirement 
centers, and church groups. 

• The consultation and education staff also 
offered its services to bilingual community 
workers and to agencies involved with Asian 
and Hispanic groups. 

SPECIAL BARRIERS OR SUPPORTS 

This study was a true cooperative effort 
between the board and center staff. The center 
director worked closely with the board chairman 
and was strongly supportive throughout the study. 
The major barriers were the 6-week limit on the 
rseds assessment and time constraints on board 
members, most of whom held full-time jobs. 

RESOURCES AND COSTS 

Board members contributed nearly 100 hours 
for planning and interviews. The research and 
evaluation director and her assistant were 
assigned nearly full-time for analysis of the 
questionnaires. CMHC clerical staff assisted. The 
study was financed out of center funds. 

ADDITIONAL COMMENTS 

This 2-month activity served a number of 
purposes. It confirmed both board and staffs 
feelings about the needs of certaiti populations in 
Alexandria. It also confirmed their feelings about 
center goals for the next year and was a learning 
experience for board members. Their knowledge 
uf center services and staff capabilities was 
vastly expanded and relationships between center 
staff and board were solidifiec,. 



29. Mental Health Advisory Board Evaluation of a 
County Mental Health Department 

Ed Diksa 

Sonoma County Mental Health Board 



SUMMARY 

Each county mental health advisory board 
(MHAB) in California is charged with providing 

For further information write Ed Diksa, 6924 
Gravenstein Highway, CoUtl, CA 94928, (707) 795-4401 
(home phone), also c/o Sonoma County M/H Board, P.O. 
Box 1412. Santa Rosa, CA 95402. 
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citizen and community involvement into the 
county's mental health plan for the upcoming 
year. The hoard is also charged with approving the 
**process** by which said plan was adopted, i.e.. 
ensuring that the plan w<js the result of careful 
reasoning after assessing county needs and 
acquiring and responding to information from all 
relevant sources* 
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TYPE OF ORGANIZATION 

The MHAB is a State-mandated body composed 
of 17 members appointed by the county super- 
visors. A majority of the total membership must 
represent the general public and a majority of 
these members must represent consumer interests 
(either recipients of service:> or members of 
clients' families). The remaining members must 
represent the various disciplines within the 
mental health field* 

EVALUATORS OR MONITORS 

Our hoard consisted of six professionals working 
in the private sector and/or teaching in mental 
health related disciplines; five .iiembers of the 
general public (housewives, students, business 
people); and four consumers (one present user of 
outpatient services, one past user of inpatient and 
outpatient services, and two parents of con- 
sumers). 

REASONS FOR THE EVALUATION 
OR MONITORING 

Our yearly evaluation is mandated by State law. 
In addition to monitoring the mental fiealth 
department, the board also serves as a channel of 
communication between the community and the 
department. The particular problems addressed in 
the evalu?»*:ion were poor staff morale and 
functioning due to long-standing dissatisfaction 
with the director and the need to review the 
handling of state mandated budget cuts. 

LEVEL OF PARTICIPATION 

Since the problems facing the department were 
very different from previous years, the board was 
forced to treat them in vastly different ways. To 
address the fiscal problems, weekly meetings 
were held with the administrators and program 
chiefs. This enabled board ,7iembers to determine 
the magnitude and areas of cuts the department 
was planning. Community meetings were also I.ela 
to raise relevant issues. Board members were then 
held to review the various ideas and data and to 
form a list of recommendations. 

The problem with the department director 
presented a much thornier issue since board 
members did not feel it was their place to become 
involved in personnel matters. The only avenues 
of intervention open to the board were issues of 
morale and quality of service. Meetings were held 
with staff of all levels, i.e., clerical, clinical, 
program directors, and other administrators. 
Numerous phone calls from staff who were unable 
to attend the meetings were also received. Board 
members then met as a group to assemble the 
findings. Once members had arrived at their 
conclusions, they were in a quandary as to how to 
proceed. The decision was to take a low profile 



since the department's internal turmoil had 
already received considerable attention in the 
press. Members went privately to each county 
supervisor and apprised them of the board's 
findings, suggesting that something needed to be 
done quickly. 

TARGET OF EVALUATION OR MONITORING 

This study encompassed both the fiscal 
problems of the entire county mental health 
department, as well as issues concerning the 
department head. The name of the agency is the 
Sonoma County Mental Health Department. The 
county has a population of about 300,000. It is 
composed of several suburban cities as well as 
widely separated and isolated rural areas. The 
administrative structure of the department 
includes a director, an assistant director, and a 
number of program managers who are in charge of 
specific program components. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The issues investigated, were twofold: (1) the 
board's response to budget cuts and (2) its 
response to the sinking morale and quality of 
services due to friction between the staff and the 
director. The board's primary concern in both 
issues was the quality of service. The budget cuts 
proposed were 10-15 percent of total program 
funds. The board felt that with cuts this size the 
department might be forced to restructure itself, 
rather lhan simply pruning some of its branches. 

With respect to the second issue, board 
members were interested in finding out about the 
modes of communication between the staff and 
the director, as well as the effect this rela- 
tionship has had on the manner in which services 
were being delivered. 

TECHNIQUES USED 

Virtually the entire MHAB was involved in this 
task. However, a core of five or six members was 
much more intensely involved than the rest. Since 
the board's mandated task is to get information 
and ideas from the comnr.unity, various com- 
munity groups were contacted and the issues of 
the budget cuts were placed before them. These 
community groups were then asked to react. 
Board members also visited the various program 
elements to learn how staff felt the proposed cuts 
would affect them, and if they felt that there 
were ways to alleviate some of the impending 
stresses. Finally, meetings were held with 
department and county administrators to discuss 
the board's proposals. In dealing with the issue of 
morale, interviews were held with nearly all 
levels of department staff, as well as with other 
county departments that had working relation- 
ships v.ith the mental health department. 
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FINDINGS OF EVALUATION OR MONITORING 

The board's major findings concerning budget 
matters were: 

1. Nearly all program elements were already 
"cut to the bone** from previous reductions; 

2. Administration and inpatient services had 
been less affected by prior cuts than had 
other elements; 

3. There was massive support for the 
continuance of outreach services in all 
oi'tlying districts (both the department and 
county administration had proposed closing 
all outreach offices as a means of meeting 
necessary reductions); 

^. Staff had mininrial knowledge of the budget 
situation and this seemed to be increasing the 
amount of tension in an already bad situation. 

Concerning staff problems with the director, the 
board's findings were: 

1. Staff felt they received inadequate support 
and information from the administration; 

2. Administration paid no attention to staff 
suggestions; 

3. Staff felt that the situation had gone on past 
the point of possible reconciliation. 

RECOMMENDATIONS 

The following fiscal recommendations were 
presented to the mental health department and to 
the county board of supervisors: 

h Computerize fiscal services and enhance debt 
collection activities; 

2. Develop computer system to handle client 
records while maintaining client privacy; 

3. Contract with private labs for testing 
services; 

Contract for clinical services in those 
instances where they could be done more 
efficiently while retaining the quality of 
services; 

5. Reduce paperwork and streamline adminis- 
trative procedures; 

6. Initiate self-help and peer counseling groups 
to relieve strain on development staff; 

7. Place greater emphasis on short-term 
therapy for outpatients; 
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8. Provide a community development team to 
organize local community support ana 
resources in the event that future cuts might 
necessitate departmental withdrawal from 
the outlying communities. 

Concerning the problem with the director, the 
board's findings were reported to the county 
supervisors. The only recommendation was that 
the situation needed to change in the very near 
future. 



STEPS TO ENSURE IMPLEMENTATION 

To achieve the recommendations concerning 
.ne budget, approval of the county plan was 
delayed for several months. Without this approval 
the department was in danger of losing all State 
funding. While the department did not implement 
all of the board's recommendations, members felt 
that the major concerns had been addressed. 

T^'e fina! decision about the director was left in 
the hands of the county supervisors since they 
were the only ones with the authority to take any 
action. 



EXTENT OF IMPLEMENTATION 

The board of supervisors reclassified the 
position of mental health director so that the 
director no longer needed to possess a medical 
degree. This was done under the guise of economy 
with the rationale being that a professional 
administrator's salary could be significantly less 
than that of an M.D. This avoided civil service 
wranglings and other political fallout that would 
have occurred had the director been removed in 
any other way. 

Budget cuts were made with an eye towards 
community needs. The principal goal of main- 
taining needed outreach services was attained. 
Cuts came from all sectors of the program rather 
than from one program component. This left the 
county with a bare minimum of coverage in nearly 
all areas. It is the board's hope that further 
service cuts can be avoided if some of the board's 
longer range recommendations are put into effort. 



SPECIAL BARRIEPS OR SUPPORTS 

The factors that made this project more 
difficult were: 

• The need to assimilate countywide data 

• Time constraints of board members and staff 

• Information provided by the administration 
was constantly changing. As a result, the 
board had no firm figures to rely on. 

95 

104 



The major supports were: 

• A unified, dedicated board 

• Support (if not always agreement) from line 
staff 

• Community support for the programs 

• Encouragement from several county super- 
visors 

• Contact with the state MHAB project of^'ice 
(a State level office designed to assist local 
MHABs) 

RESOURCES AND COSTS 

Overall, a minimum estimate would be about 
1,000 hours of ^^olunteer time over a 3-month 
period. There are no paid staff attached to the 
board. The line staff rnet with board members 
either as part of their work or in their off hours. 
Board members are reimbursed for expenses upon 
submission of the proper forms (not all members 
chose to submit such forms). 

ADDITIONAL COMMENTS 

In general, board members were satisfied with 
the job they did, considering the complexities of 



the situation. However, members felt that it 
would have been of great help to have clarified 
the support system from the start, to have 
developed a well defined organizational plan, and 
to be clearer and more forceful about the need 
for reliable d (both programmatic and 
budgetary). 

The process itself seemed workable and capable 
of fulfilling the needs for involving citizens in 
discussions and recommendations. However, the 
meetings could have been run in a more stream- 
lined manner in order to achieve comparable 
results in a shorter period of time. 

Finally, and most importantly, members found 
it difficult to keep a focus on the overriding issue 
of service quality when confronted by economic 
and personality issues. The constraints imposed by 
budget cuts threatened to change the focus to 
more practical matters and to suggest certain 
short-term solutions that could potentially 
weaken the long-tenn viability of humane and 
just service delivery. There seemed to be an ad- 
ministrative attitude that there is only room for 
ethical discussions in times of surplus. To argue 
for ethical, humane treatment of clients in times 
of economic crisis was an uphill battle that board 
members felt they had to undertake. It gave 
clarity to the goals and engendered in members a 
sense of power that was needed in order to 
confront the situation in which the board found 
itself. 



30. Volunteers Monitor State Institutions 
Serving a Metropolitan Area 

Jan Holcomb, Ann Neradj and Karen Helfrich 

Merited Health Association of Greater Chicago 



SUMMARY 

The Mental Health Association of Greater 
Chicago (MHAGC) site visitation committee was 
initiated in the fall of 1977. Since its inception, 
the purpose of the committee has been twofold: 
(1) to monitor inpatient care in vhe Region II 
Illinois Department of Mental Health and 
Developmental Disabilities (DMH/DD) facilities 
on an ongoing basis and (2) to provide current 
information for MHAGC public policy and 
advocacy activities. After each site visit, the 
findings and recommendations are written up and 
submitted to the facility superintendpot, regional 
administrator, and DMH/DD director. Implemen- 
tation of site visit results has included use of the 
media, lobbying, and testifying before the State 
legislature. 



For further information write Jan Holcomb, 20630 W. 
Exeter, Kildeer, Illinois 60047, (312) 438-2540. 
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TYPE OF ORGANIZATION 

The MHAGC is the largest metropolitan 
chapter in a network of 850 mental health 
association chapters whose goals are to promote 
mental health, to prevent mental illness, and to 
improve care and treatment of the mentally 
disabled. Programs which achieve those goals 
include public education and information, public 
policy, advocacy, and patient services. Tne site 
visitation committee is one of several program 
committees appointed by the MHAGC board of 
directors. 

EVALUATORS OR MONITORS 

Site visitation volunteers have diverse back- 
grounds. One-half of the team members are 
homemakers with professional degrees in 
teaching, nursing (two in psychiatric care), social 
work, speech therapy, and sociology. Currently 
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employed volunteers include a public relations 
specialist, a commodities broker, an artist, and a 
travel agent. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The site visitation committee was initiated for 
several reasons: (1) a MHAGC bo?»rd member, who 
had served as an ombudsman in another State, 
believed that the monitoring function could be 
performed by trained volunteers; (2) the concept 
was in keeping with the purpose of the organiza- 
tion; and (3) although periodically surveyed by the 
Joint Commission on Accreditation of Hospitals 
and the Illinois Department of Public Health, no 
independent advocac' organization monitored 
care and treatment in DMH/DD facilities on a 
continual basis. Furthermore, two of the six 
DMH/DD facilities were under fire in the press 
for various violations. 

LEVEL OF PARTICIPATION 

Volunteers have been involved in all aspects of 
the site visitation program including the follow- 
ing: (1) negotiating the working agreement with 
DMH/DD that resulted in an open door policy ft r 
the committee in all facilities; (2) developing site 
observation Corms, procedures for site visits, and 
report writing format; (3) developing guidelines 
for training and inservice; and (4) developing job 
descriptions for volunteers and volunteer/asso- 
ciation agreements; and participating in foUowup 
with DMH/DD officials. 

TARGET OF EVALUATION OR MONITORING 

The focus of the monitoring activity has been 
the six State-operated adult inpatient facilities 
for the mentally ill serving the greater metropol- 
itan area (DMH/DD Region II). This region alone 
generates 15,900 admissions per year to these 
State-operated facilities. Three of the region's 
facilities serving the mentally ill are designated 
for acute care. The mission of a fourth facility 5 s 
research and training, and the two most outlying 
institutions serve as long-term care facilities, 
although both receive acute admissions from their 
immediate catchment areas. 

After surveying the adult facilities for 3 years, 
the committee members were trained to survey 
the three children and adolescent centers. Two of 
ther>e units are within hospitals for acute 
illnesses, while the third is a separate facility on 
the grounds of a hospital for acute illnesses. 

Finally, the successes of the site visitation 
committee prompted the members to develop the 
program statewide. Recently, training was 
offered to volunteers representing three of the 
State's other regions. The goal is to have 
association volunteers, vyhere chapters exist, 
monitor all DMH/DD-operated inpatient facilities, 
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PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Site visits are comprehensive and cover the 
following facility issues: (1) goals, policies, and 
procedures; (2) physical plant and housekeeping; 

(3) staffing— composition and attitudes; 

(4) intake, referral and discharge procedures; 

(5) intervention modalities (psychotherapeutic 
services, vocational, etc.); (6) group living; 

(7) residents' rights; (8) utilization of community 
resources; (9) dietary service; (10) laundry; and 
(11) education model (children and adolescents). 
Particular attention is paid to the following 
persons: (1) the sensory impaired; (2) those not 
guilty by reason of insanity; (3) the difficult to 
manage; and (4) children and adolescents. 

TECHNIQUES USED 

Volunteers use observation fornns to record the 
information gathered from onsite facilities and 
unit visits. InformatiOii is obtained through 
personal observations and interviews with 
patients, staff, and administrators. The informa- 
tion, compiled in a narrative report with specific 
recommendations, is submitted to the facility 
administration for a response. 

The observation format was developed by 
volunteers and MHAGC's progrann director and is 
revised as needed. Resources that were utilized 
and adapted for the connmittee's needs included: 
Citizen Evcdmtion of Mental Health Services: An 
Action Approach to Accountability, the National 
Menta' Health Association's Community Mental 
Health Center Visitation Handbook, the observa- 
tion format of the Joint Conn mission on the 
Accreditation of Hospitals, and professional 
sources. 

FINDINGS OF EVALUATION 
OR MONITORING 

In the past 5 ysars, the site visitation commit- 
tee volunteers surveyed six facilities under the 
Icadcship jf two State mental health department 
directors, three regional administrators, and 
several institutional superirttendf=^nts. Visitation 
findings reflect significant variations in physical 
plant, numbers and quantity of staff, clinical 
commitment, and leadership. For example, the 
two oldest facilities— built in the late 1800s— 
present different physical plant problems from 
those facilities constructed in the 1960s. The 
team also found that staff with 20 to 25 yeais of 
service have different att-itudes regarding 
patients* rights and treatment programs than 
staff who are recent graduates of social work, 
psychology and activity therapy programs. 
Moreover, a business manager mth 20 years of 
public service has a different leadership style 
from a clinical psychologist with experience in 
direct care. 
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Even within facilities, volunteers reported 
significant variations. Committee members 
observing two units in the same facility having 
the identical physical layout, treating patients 
with parallel clinical and demographic profiles, 
and possessing the same staffing pattern, might 
find one unit clean and tidy, decorated with 
artwork and plants, and with current clinical 
documents and expectations clearly posted. The 
other unit might offer none of those charac- 
teristics. Perhaps the most unique contribution of 
the committee is systemwide observation over 
time. 

RECOMMENDATIONS 

A sample of the types of recommendations 
presented to the relevant facility /unit include the 
fcUowin-j: 

• Facility*>specific recommendations .— Pa- 
tients* rights will be displayed clearly and 
permanently by the nursir.g station; 
bathrooms will have soap and toilet paper; 
the gymnasium will be available on a regular 
and predictable basis; restraints will be 
removed from individual beds when not being 
used; contact paper on bedroom windows 
should be used as an alternative to ripped and 
filthy drapes hanging by a couple of hooks. 

• Intrafacility recommendations .—Units 
performing a good job in linking patients with 
community clinics should offer training to 
other units that are not assuring continuity of 
care; ••successfur discharge staff should help 
••failing** discharge staff develop techniques 
and procedures to strengthen patient/ 
community connections; units using a 
behavior modification orientation should 
clarify their •'overt systems'* guidelines to 
reward appropriate behaviors to be certain 
that patients understand exactly what they 
have to do for each reward, for example, 
earning a grounds pass. 

STEPS TO ENSURE IMPLEMENTATION 

The visitation reports are addressed to the 
facility superintendent with copies sent to 
appropriate regional and central office staff. 
Upon receiving the facility superintendent's 
response to the report, the team conducts a 
followup visit to determine whether or not 
recommendations have been carried out. These 
visits are goal specific and usually shorter in 
length. Reports from the followup visits nay be in 
the form of a letter or short memo reporting the 
team's findings. Typically, some improvement is 
found on a foUowup visit in the areas for which 
the facility has resources. 

Another step ensuring implementation is the 
monthly meeting of the b*te visitation committee 
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Chair, the children and adolescents subcommittee 
chair, and mental health association staff with 
the Region II administrative staff. This meeting 
addresses facility-specific a«; ♦veil as regionwide 
lasues. 

In a Jdition, Mental Health Association of 
Greater Chicago (MHAGC) volunteers and 
program staff have worked diligently to establish 
a rapport with State legislators and their staff. 
For example, in the spring of 1982, House 
Republican staff contacted the MHAGC to assist 
them in developing the agenda for public hearings 
regarding the DMH/DD appropriations; in early 
winter 1983, the Illinois Commission on Mental 
Health and Developmental Disabilities did the 
same. Moreover, site visitation commitee 
members have offered testimony to legislative 
cor..mittees on public policy issues affecting the 
care of the mentaUy ill in Illinois. At the urging 
of the site visitation committee, the mental 
health aj^sociation in Illinois worked through its 
lobbyist to introduce a bill recommending 
upgrading and/or clos ng of certain State 
facilities. 

The committee also shares information with the 
Illinois Guardianship and Advocacy Commission 
and its human rights ojthorities (HRA's). These 
HRAs have investigative and subpoena powers but 
do not monitor inpatient care on a routine basis; 
thus, it is beneficial to both site visitation 
members and the HRAs to maintain close contact. 
The site visilation committee has provided 
training both for paid staff and volunteers of 
several of the HRAs in Illinois. 

After 5 years of visitations, reports, recom- 
mendations, and followup, committee members 
were still frustrated that s^'stem changes had not 
been made at the executive and legislative levels. 
It seemed necessary to call public attention to 
these problems in order for the decisionmakers to 
address them. When reports from the Chicago 
Sun-Times requested a comment on one small 
recurrent problem, the committee suggested that 
they consider addressing the problems of the 
larger mental health system utilizing MHAGC's 
expertise. After a briefing session and an editorial 
decision to pursue problems in depth, site 
visitation volunteers and MHAGC's progran 
director accompanied news reporters to each 
facility and provided contacts to them for other 
aspects 0"^ their proposed series. 

EXTENT OF IMPLEMENTATION 

At the facility level, the committee has found 
that the majonty of the facility superintendents 
have attempted to correct identified deficiencies 
within existing fiscal limitations. For example, 
housekeeping practices have been improved in 
many instances; on most units, information 
orienting patients to time, date, rights, and 
activities are now more likely to be in place; 
access to activities and the addition of activity 
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personnel have increased; facility administrators 
have helped certain unit service supervisors 
manage their units more effectively; and patiencs 
in most units now have access to their bedrooms 
for rest periods (in the past bedrooms were locked 
in the morning and remained so until bedtime). 
The amount and quality of staff interaction and 
treatment services that patients receive are 
ongoing concerns for the committee and require 
constant monitoring. 

The Chicago Sun-Times investigation, referred 
tc earlier, resulted in a six-part series entitled 
••Breakdown: Mental Health in Crisis.'' The series 
accomplished its purpose in that either the 
Governor or his staff personally visited each 
Region II facility. The Governor also appointed a 
task force to address the issues outlined in the 
series and solicited the site visitation commit- 
tee's ideas in the planning process. For example, 
the committee met with the Deputy Governor and 
presented three specific recommendations 
concerning system level reforms. All three 
recommendations were acted upon; the most 
significant action exempted DMD/DD from an 
across-the-board 4-perc^nt reduction due to 
affect all code departments. Furthermore, the 
Deputy Governor invited the committee to review 
the 1982-83 budget prior to submitting it to the 
legislature. Finally, on the legislative front, the 
Mental Health Association of Greater Chicago 
introduced a bill which reflected thf=t recom- 
mendations of the site visitation committee. The 
bill was passed in the Senate but fai!od in the 
House. 

SPECIAL BARRIERS OR SUPPORTS 

Although the site visitation committee has had 
a highly dedicated core of volunteers, normal 
attrition (i.e., /noving away, employment, etc.) 
has limited expansion of its monitoring activities 
to include the private sector. In addition, though 
the professional volunteer group makes evening 
and weekend visits, this employed group is small 
and often without transportation; thus, nonweek- 
day visits to the outlying facilities are almost 
impossible and visits to the closer facilities ire 
not as frequent as they should be. 

Another problem has been the amount of time 
required to compile the site visit report. Written 
observations must be collected usually from three 
to four volunteers who may be busy with other 
commitments; therefore, it is difficult for the 
team leader and the association to complete the 
report within the required time period. To resolve 
this problem, inr^aller teams have been ignated 
to cover fewer units per visit. 

Other barriers are unique to a large state 
bureaucracy including: the institutionalization of 
certain staff; an employee union that protects 
abusive or incompetent staff; and the "refer it for 
further study to a committee** approach to 
problemsolving and otner issues. When necessary, 
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the media has been used to publicize these 
problems and, therefore, to put pressure on 
elected officials. 

Particular supports to this committee include 
the following: a program director who is know- 
ledgeable in, and committed to, public policy and 
the volunteer-scaffed site visitation program; a 
board of directors who support the purpose of the 
sice visitation committee, i.e., to be the **eyes and 
ears** of the association; and the public and 
private sector's involvement in training com- 
mittee members. 

RESOURCES AND COSTS 

Volunteers pay their own expenses (i.e., 
transportation, daycare, meals, etc.), anc* there 
are no recruitment costs since volunteers and 
staff recruit other potential members. (The Junior 
League of Chicago is a primary recruitment 
source.) In 1981, 10 volunteers donated 720 hours 
or 120 days to the site visitation program. This 
total includes only the actual visits and report 
preparation time — not time spent in meetings, 
lobbying, testifying, and traveling. 

MHAGC staff is also provided to the site 
visitation program. Approximately 50 percent of 
two staff positions— the program director and the 
program secretary— are allocated to site visita- 
tion and public affairs. Other indirect costs that 
are covered by the association include phone, 
supplies, postage, and staff travel reimbursement. 

ADDITIONAL COMMENTS 

In addition to its achievements in the areas of 
care, treatment and human and civil rights of the 
institutionalized mentally disabled, the site 
visitation committee has greatly enhanced the 
image of the Mental Health Association of 
Greater Chicago as a public policy resource and 
advocate. In the spnng of 1982, the site visitation 
program was recognized by the Volunteer Action 
Center of Comprehensive Community Services of 
Metropolitan Chicago for its innovative use of 
volunteers in accomplishing agency purpose. 
Furthermore, at the spring 1982 DMH/DD annual 
mental health luncheon, the site visitation 
committee received one of two awards selected 
by the director for unique and innovative 
programs benefiting the mentally ill. And finally, 
the site visitation committee has been recognized 
by the National Mental Health Association as an 
exemplary monitoring effort, and requests are 
received from MHAs across the United States 
that want to replicate the program. 

The success of the program involves several key 
elements including trained and committed 
volunteers as well as the capacity for self- 
evaluation that leads to continued program 
improvement. The intent of the program was not 
to rely on sensationalism through the release of 
public reports but to effect change through 
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ongoing monitoring. Even in providing information program was maintained and the open door policy 
to the media, the integrity of the site visitation with the state DMH/DD was not jeopardized. 



31. Mental Health Board Members Allocate Resources 
Using a Computerized Decision Tool 

Jean A. Abruzzino and John D. Corrigan 

Franklin County Mental Health Booj^d 



SUMfvlARY 

Each year the planning committee of the 
Franklin County Mental Health Board undergoes 
an extensive evaluation process to assess need, to 
determine priorities for services and recipient 
groups, and to recommend allocations for the next 
fiscal year. Because of the complexity of multiple 
funding sources, as well as the multiple needs of 
individual agencies, planning committee members 
use a computerized de::isionmaking tool when 
developing final recommendations for funding 
priorities. 

TYPE OF ORGANIZATION 

As prescribed by State law, the Franklin County 
Mental Health Board is composed of fifteen 
volunteers representing a variety of occupations 
and interests. Two-thirds of the members are 
appointed by the county commissioners, and 
one-third is appointed by the director of the Ohio 
Department of Mental Health. Duties of the board 
include planning, funding, evaluating, and 
overseeing the development and creation of 
mental health services located in the community. 
The board does not operate services directly but 
instead uses a purchase-of-service mechanism to 
contract with private, not-for-profit agencies. 

EVALUATORS OR MONITORS 

Many bocrd responsibilities, including evalua- 
tion, fall under the purview of the planning 
committee. The committee is made up of seven 
board members, one representative of the 
chairpersons of Franklin County's seven catch- 
ment areas, and one representative of the seven 
catchment area executive directorc. Members of 
the planning committee come from varied 
backgrounds Including physical medicine, politics, 
labor management, higher education, and 
business. Professional board staff provide 
members with technical expertise and assistance. 



For further information write Jean A. Abruzzino, 
Director of Planning, Franklin County Mental Health 
Board, 447 East Broad Street, Columbus, OH 43215, 
(614) 224-1057. 



REASONS FOR THE EVALUATION 
OR MONITORING 

Under Ohio law, the county mental health 
boards are responsible for community menta) 
health planning. The planning process develoj-c-d 
by the Franklir County board occurs annually, 
several monthf, prior to the beginning of the next 
fiscal year. In tre light of shifting needs and 
resources, objective, ongoing review of priorities 
and funding for servicco is an essential part of 
targeting services and maximizing dollars. 
Through this process, accountability to the 
community and to funding sources is maintained. 

LEVEL OF PARTICIPATION 

During the planning process for fiscal year 
1982, committee members were actively involved 
in evaluating needs data, reviewing projections, 
and voting on systemwide priorities for services 
and specialized populations. Information and ideas 
from board staff members and the seven catch- 
ment areas were also utilized. The committee 
members took into consideration the services that 
were available outside the community mental 
health system, as well as agency funding from 
sources other than the mental health board, in an 
effort to coordinate a balanced mental heclth 
delivery syster 

TARGET OF EVALUATION 
OR MONITORING 

The committee is charged with planning mental 
health services on a countywide basis. Approxi- 
mately 870,000 persons reside in Franklin County; 
as such, the mental health svstem is complex, 
consisting of seven catchment areas and 34 direct 
service providers. Committee members plan for a 
continuum of residential, day treatment, out- 
patient, consultation and education, and emer- 
gency services for all age groups. Programs can 
be either specific to catchment area or county- 
wide. In addition to mental health services, 
committee members also plan for countywide 
drug abuse services. Planning is *'Hus broadly 
bar 3d and targeted at many different program 
areas, agencies, and recipient groups. The 
following partial list of services illustrates the 
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diversity of programs that committee members 
must address: supervised apartment living; adult 
foster care; services for the severely emotionally 
disturbed child and adolescent; a halfway house 
for runaway youth; general counseling services; a 
battered women's shelter; and residential homes 
for aftercare and drug abuse clients. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The committee members, during the annual 
planning and evaluation process, consider revenue 
projections from all agencies, as well as an initial 
estimate of funds available from Federal, State 
and county sources. Projected services are locked 
at in relation to needs data, including past 
purchase-of-service, agency budgets, and census 
demographics. Priorities for mental health and 
drug abuse services are evaluated in terms of the 
function they are intended to serve (i.e., crisis 
stabilization, growth, sustenance), and the 
environment in which they are delivered (i.e., 
protective, supportive, natural). In addition, the 
committee must agree on the minimum resources 
to be provided to particular recipient groups (i.e., 
children, adolescent, adult, elderly, aftercare). 
Both long and shortrange planning are incor- 
porated into this process. Consideration of 
shifting needs and revenues affect more immedi- 
ate goals, while the availability of historical data 
allows committee members to evaluate progres- 
sion toward long term goals. 

TECHNIQUES USED 

In order to evaluate resource allocetion issues, 
the planning committee is assisted by a computer 
tool known as the Goal Programing Model (GPM). 
The GPM allows consideration of a variety of 
complex funding scenarios, permitting planners to 
evaluate the impact o^" priority decisions on all 
aspects of the system. Through a weighted voting 
procedure, initial priorities are determined by 
committee members and then entered into the 
computer. Based on these priorities, the GPM 
provides a "solution" showing the distribution of 
dollars that would occur across programs and 
recipients. Committee members then evaluate the 
solution, review priorities, and compute new 
solutions until satisfied with the distribution 
pattern. The following pieces of information are 
incorporated into the computer model: 

Services .— In the GPM, services are divided 
initially into either mental health or drug abuse 
disabilities. For each type of disability, services 
are further defined by the functions (i.e., crisis 
stabilization, growth, sustenance, consultation, 
and education) specified in the balanced service 
system model adopted by the Joint Com.nission on 
Accreditation of Hospitals. Most service functions 
can be delivered in one of three service environ- 
ments—protective (inpatient), supportive (the 
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mental health system), and natural (the home, 
school, or place of work). When a service function 
is matched with z specific service environment, a 
cell is created. Within each cell, services are 
further specified in the CPM by five charac- 
teristics: (1) type of service (e.g., residential or 
day program), (2) required supplemental service 
(e.g., case management), (3) service delivery 
agency (e.g., one of contract cr subcontract 
agencies), (A) recipient (e.g., child, adult) and (5) 
recipient's catchment area of residence. 

Need .— The GPM coritains the previous year's 
service delivery level for all levels of each 
variable just described. Data from a countywide 
demographic projectio*^ of ideal service levels are 
also incorporated. 

Finances.— Tlie GPM has three pieces of 
financial information for services delivered by 
each agency: total revenues available, the sources 
of revenues (i.e., community mental health funds. 
Title XX, specific Federal fuiids), and the cost per 
unit of service. When several agencies offer the 
same service, the GPM selects the lower unit 
cost, thus encouraging efficient service delivery. 

Priority and recipient group constraints . — 
Committee-determined priority weights and 
minimum funding levels for specialized popula- 
tions are included in the model. Within the 
computer pogram, priority weights allow the 
value or importance of services to be reflected in 
their funding levels. 

The Goal Programing Model (GPM) uses all of 
the above information to accomplish three 
criteria: (1) minimize unmet need based on 
priorities given to those needs, (2) accomplish 
criterion (1) in the most financially efficient 
manner possible, and (3) accomplish criteria 
(1) and (2) within the tp'^hnical constraints 
required for a feasible allocation pattern. Thus, 
each time the computer buys a unit of service, it 
is looking at whither the service is needed, how 
important the f ervice is, where the service can be 
purchased at the lowest cost, and whether it is 
feasible to buy it. 

A set of reports i produced automatically each 
time a model is run on the computer. Each report 
has been developed to provide committee 
members with a different perspective on the 
distribution of funds according to a particular 
GPM solution. For example, table 1 shows the 
spread of total dollars and units across service 
function and environment. The effect on service 
distribution due to committee changes in priority 
weightings is best tracked with this report. 

FINDINGS OF EVALUATION OR MONITORING 

With the aid of a computerized model, the 
process of ranking by priority involved committee 
members in planning for expanding service needs 
during a time of shrinking revenues. Recognizing 
that resources could not meet all the mental 
health needs in the county, priorities were spread 
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across the system in such a way as to ensure that 
the greatest need was nf^et at the lowest cost. 
After assigning weights to each nnental health 
service, the services were ranked. Mental health 
priority rankings shifted sonnewhat fronn the 
previous year, but rankings for substance abuse 
services remained constant. For 1982, the three 
top-ranked mental health services included crisis 
stabilization in a supportive environment, growth 
in a supportive environment, and growth in a 
natural environment. The committee also 
identified as recipients three priority groups: 
children, adolescents and aftercare clients. 

RECOMMENDATIONS 

Funding for all agencies was recommended at 
90 percent of 1981 mental health board levels, 
with the exception of the three recipient priority 
groups cited above. Funds beyond the 90 percent 
level were to be distributed among the priority 
areas in the following way: day treatment and 
outpatient services for children, 0-12 years; day 
treatnnent and outpatient services for adoles- 
cents, 13-17 years; and residential, day treat- 
ment, and outpatient services for aftercare 
clients, age 18 years and older. 

STEPS TO ENSURE IMPLEMENTATION 

Each year funding recommendations are 
presented at public hearings to allow agency and 
community feedback prior to submittal to the full 
board. Following board approval, the final 
package is routed to the finance committee to be 
used in writing individual agency contracts for 
purchase-of -service units. Priority recommen- 
dations for services are reflected in the contracts 
since they delineate the nunnber of units of 
service to be purchased from each program area, 
as well as the unit cost at which the agency is to 
be reinnbursed by the board. In addition, recom- 
mendations affect ongoing planning committee 
decisions regarding the development or termina- 
tion of programs. 

EXTENT OF IMPLEMENTATION 

Due to funding reductions and uncertainties at 
the Federal and State levels this past year, 
recommendations for allocations beyond 90 
percent of 1981 mental health board funds for 
children, adolescent, and aftercare clients could 
not be implemented. Although all services are 
currently funded at this 90-percent level, the 
priority weights and recommendations developed 
during this procer 3 continue to influence planning 
decisions. 

SPECIAL BARRIERS OR SUPPORTS 

The primary barrier to the 1982 planning 
process was the inability to project accurately 



what revenues the commi^tee could use in its 
decisionmakinc pror^ss. At the Federal level, the 
extent of proposed reductions in Title XX funding 
was unknown. In addi^ n, the financial impact of 
moving from Federal categorical grants to State 
block grants was difficult to assess. At the State 
level, severe economic conditions and recurring 
fiscal shortfalls slowed the development of a 
State budget. A series of interim State budgets 
made planning a virtual impossibility. Throughout 
the current year, periodic reductions in the State 
mental health budget have necessitated conserv- 
ative planning and funding. 

The Goal Programing Model (GPM), however, 
serves as the greatest single source of technical 
assistance for committee members. In spite of its 
complex appearance, committee members have 
found it a useful and understandable tool for 
analyzing different funding scenarios and 
evaluating tha impact of different decisions. The 
GPM has allowed board members greater control 
over the complex decisions involved in mental 
health planning. The actual reports produced by 
GPM "solutions** were developed with committee 
member input. These reports provide funding 
inform-'*'..ion on a variety of levels and are 
produced in formats designed to summarize data 
clearl>. The GPM has aaded a degree of objec- 
tivity to what was previously a highly politicized 
decisionmaking process. The level of current 
planning effort would be impossible without this 
computer support. 

RESOURCES AND COSTS 

Consulting and computer support are provided 
through board-approved contracts with the Ohio 
State University. In 1981, contract costs amount- 
ed lo approximately $58,000. This level of funding 
represents the investment in initial development. 
Future annual costs, however, will be lower, 
primarily covering upkeep and maintenance 
activities. The entire planning process for 1982 
occurred over a 3-month period that began in 
April of W81. Committee members and board 
staff met weekly throughout April and May. 
Public hearings were held in early June, and final 
recommendations were approved by the board two 
weeks later. 

ADDITIONAL COMMENTS 

The GPM is undergoing constant development. 
Board and contract staff have expanded the 
model's original capabilities and are now using the 
computer to create scenarios that can project the 
distribution of funds in the event of a sudden, 
systemwidc financial disaster, and that can 
predict the need for additional levy monies. 
Report formats continue to be improved based on 
continuing updating by committee members. The 
Goal Programing Model (GPM) retains its ability 
to adapt to new planning techniques and needs — a 
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quality that is particularly important in a time of flexibility needed to plan effectively for the 
funding uncertainties. A decision support tool of mental health needs of Franklin County residents, 

this kind provides board members with the 



Table 1 

DISTRIBUTION OF FUNDS BY SERVICE FUNCTION AND ENVIRONMENT 

Summary of Money Allocated by Service and Environment 
Grand Total of Money Allocated; $20,600,316.77 

BSS Reports 







Proiective 


Supportive 


Natural 


Crisis Stabili- 
zation 


Units 
Dollars 


5576 
$254,049.00 


45532 
$2,555,146.51 


3566 
$148,132.00 


Growth 


Units 
Dollars 


2168 
$37,456.00 


288984 
$12,701,837.85 


12221 
$280,501.38 


Sustenance 


Units 
Dollars 


489 
25.711.00 


38800 
$1,867,075.64 


16815 
$259,534.00 


Supplement 


Units 
Dollars 


2024 
$93,220.00 


21781 
$974,120.94 


7047 
$253,607.48 


Consultation 
and education 


Units 
Dollars 






30467 
$1,449,925.00 



32. Citizens' Group Responds to Regional Funding Needs 

Frances P. Meehan 

San Gabriel Valley Mental Healch Services, Liaison Committee 



SUMMARY 

The San Gabriel Valley Mental Health Services 
Liaison Committee found a gross disparity in the 
level of State and county mental health funds 
allocated to the San Gabriel Valley as compared 
to other regions in Los Angeles County. Facts 
were assembled, graphed and presented by a 
citizen group to the Los Angeles County Super- 
visor from this district. The result was an 
Immediate and substantial increase in funding and 
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a marked improvement in funding, manpower, and 
facilities since since action was taken in 1972. 

TYPE OF ORGANIZATION 

The Mental Health Services Liaison Committee 
(subsequently known officially as Regional 
Committee Liaison Committee or RCLC) is 
composed of lay citizens who live in San Gabriel 
Valley and who are representative of the socio- 
economic, ethnic, and occupational demography 
of this area. The role of the committee is to 
advise the director of mental health services 
regarding the needs of the region and to educate 
organizations about the services available. The 24 
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members of the committee are appointed by the 
county director of mental health services on the 
advice of the regional director. 

EVALUATORS OR MONITORS 

The members of RCLC were educators, 
community leaders, and relatives of clients. In 
1972, when these events took place, there were 
members of the League of Women Voters who had 
recently conducted a study of mental health 
services. The chairman of the RCLC, who had a 
degree in medical physiology, worked very closely 
with professional staff as well as committee 
members. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The San Gabriel Valley became a region of Los 
Angeles County three years before the study took 
place in 1972. At that time, the region— which 
represents one-fifth of Los Angeles County— had 
a grossly inadequate level of funding, insufficient 
manpower, and only one facility to meet its 
needs. All fiscal accounting was done in the 
centrp^' office, but it was obvious that, year after 
year, the San Gabriel Valley received far less than 
its fair share. The RCLC participated each year 
in evaluating the needs of the region and assisting 
in development of the plan for mental health 
services prepared by the Los Angeles County 
Department of Mental Health for submission to 
the California State Department of Mental 
Hygiene. 

LEVEL OF PARTICIPATION 

The committee functioned as an independent 
entity— separate from the county advisory board 
or county department. Thft chairperson took the 
lead by studying the pertinent parts of the 
voluminous Los Angeles county budget. A clear 
presentation including percentages and bar graphs 
were then made to the liaison committee and to 
several community organizations. 

TARGET OF EVALUATION OR MONITORING 

In 1972, the San Gabriel Valley Mental Health 
Services Region \/as one of 12 in Los Angeles 
County, but it Sf rved by far the largest popula- 
tion, 1.2 million, or 17.6 percent of the total. The 
region covers a 772-square-mile area and the 
problem of transportation was (and is) great. Even 
though there were very few private mental health 
resources in the area, the county had set up only 
one clinic in the San Gabriel mental health 
service area. This was administered by a psychi- 
atrist who reported to the psychiatric dirsctor of 
the Los Angeles County mental health depart- 
ment. He in turn reported to the Los Angeles 
County board of supervisors, composed of five 
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rnen with absolute administrative and legislative 
power and no review of their actions. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The goal of the liaison committee was to 
alleviate the severe funding limitations that had 
resulted in o dearth of nr .ntal health manpower 
and facilities in the area--both public and 
privately contracted. 

TECHNIQUES USED 

The liaison committee had as part of its charge 
the review of the mental health needs of the 
citizens of the area and the establishment of 
priorities. When annual reports repeatedly 
resulted in little or no improvement in funding for 
needed services, it was obvious there was a 
maldistribution of funds in the county for 
professional personnel and facilities. To validate 
this perception, a copy of the pertinent sections 
of the Los Angeles County budget was obtained 
and reviewed. This material was then presented to 
citizen groups in a clarified form. 

FINDINGS OF EVALUATION OR MONITORING 

State mental health funds allocated by region 
within Los Angeles County in the 1972-73 budget 
showed that the San Gabriel Valley mental health 
service area received 6.1 percent of the total for 
17.6 percent of the population, or the equivalent 
of $1.00 per capita as compared with $7.30 per 
capita in another region. The same budget showed 
that the area received 1.1 percent of county 
project priorities, and 4.5 percent of the contract 
allocations. The total per capita allowance 
(including hospitalization costs, contract costs, 
and jointly operated costs) for the San Gabriel 
Valley was $5.34 compared to a county average of 
$8.56, and a high of $18.78 in another region. 

With respect to service need, the evaluation 
and research division of the Los Angeles County 
department of mental health ranked San Gabriel 
as number one on their need index. Using 15 
variables, they reported a need of 15.31 percent. 
This meant that of the $60,000,000 projected for 
the county, the fair share for this region, would 
have been $9,186,000 instead of the $-6,613,000 
allocated. 

Needs identified by the RCLC in a cooperative 
study with the staff were: (1) four additional 
satellites or clinics, (2) personnel to replace those 
lost in the ^'freeze," (3) personnel to staff 
additional clinics, (4) contract services, (5) crisis 
homes, and (6) rhildren's services. In short, all the 
services for all ages were inadequate. 

RECOMMENDATIONS 

A report on these funding issues was made to 



the county supervisor representing the San 
Gabriel Valley. The specific request was for an 
additional $2,500,000 in order to secure a fair 
share of the budget and to fund some of the 
identified needs for the region. Previous recom- 
mendations made to the director of mental health 
services for Los Angeles County had fallen on 
deaf ears. 

STEPS TO ENSURE IMPLEMENTATION 

Brief talks using charts and graphs were r>ade 
to members of the liaibuii committee, the Los 
Angele3 County mental health advisory board, 
American Association of University Women, 
League of Women Voters, Arcadia Coordinating 
Council, the Regional Planning Council of United 
Way, service clubs, anJ others. 

Since requests to the director of mental health 
services for the county had gone unheeded, an 
appeal had to be made to the Los Angeles County 
Supervisor from the San Gabriel area. Because 
county supervisors are very powerful by virtue of 
their control over a huge budget and range of 
services, it was essential to make the strongest 
possible case. Twelve community leaders 
representing rome of the above groups accom- 
panied the chairperson of the committee to plead 
with the supervisor for a more nearly equitable 
division of State and county mental health funds. 

Following the oral presentation, copies of a 
fact sheet, graphs, and tables comparing the 12 
mental health regions with regard to population, 
area. State funds, per capita allocations, outpa- 
tient units of service, and cost per patient were 
presented to the supervisor. 

EXTENT OF IMPLEMENTATION 

The following week, the supervisor arranged for 
an additional $1,271,597 from recently released 
surplus State funds for mental health prograns in 
the San Gabriel Valley. Because the c^- .ty 
department of mental health and the board of 
supervisors do not take funding from one region to 
give to another, it would be achieved by allo- 
cating more •*new money** to the neediest areas. It 
is important to note that the administration, 
organization, accounting procedures, and State 
plan requirements have all changed markedly in 
the last ten yea. s. 



SPECIAL BARRIERS OR SUPPORTS 

The central accounting office of the county 
department of mental health did not make fiscal 
data available even to the professional staff, and 
certainly not to citizen groups. 

Positive factors included the existence of a 
group of dedicated citizens who would, for the 
most part, be classified as "housewives** in 1972. 
They had the time, interest, and persistence. 
Finally, the process was enhanced by the concise 
piesentation to a receptive supervisor of incon- 
trovertible facts in graphic form. 

RESOURCES AND COSTS 

There were no paid staff and no monetary 
resources. After 10 years it v.-uld be difficult to 
estimate the many hours of volunteer time given 
by the citizens involved. 

ADDITIONAL COMMENTS 

This is an example of remarkable "beginner's 
iuck" in having an impact on the political proce«3. 
Advice to others includes the following: 

• Present facts in clear, concise manner with 
visual materials. 

• Leave copies with officials and with the 
press, if appropriate. 

• Have a small group of respected advocates. 

• Arrange a planned presentation for one or 
two speakers. 

• Be courteous and positive. 

• Se.id an immediate followup letter. 

A lasting friendship and alliance with this 
supervisor was started at the above reported 
meeting. The chairperson has since been asked to 
serve on a number of blueribbon committees for 
health affairs and has served for two terms on the 
Los Angeles County mental health advisory board, 
two years as chairperson. Continued and remark- 
able improvement has been made in mental health 
services for the San Gabriel Valley. 
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33. A Community Service Board of Evaluation 
of Service Delivery 

Barbara Todd 

Fcdrfax-Falls Church Cornmmity Services Board 



SUMMARY 

The Fairfax-Falls Church community services 
board provides a comprehensive array of mental 
health, mental retardation and substance abuse 
services. Community service boards in Virginia 
are the agencies designed as responsible for 
administering the expenditure of public funds and 
overseeing the programmatic and administrative 
operation of these agencies. Members are 
appointed by the governing bodies of the com- 
munities 'served by each board. 

As Federal funding for mental health services 
declined, it became apparent that modifications 
to the mental health system would be required. 
With this in mind, the mental health committee of 
the Fairfax-Falls Church Community Services 
Board created a consolidation task force (CTF) to 
explore the possibility that some form of 
systemwide consolidation could effect sufficient 
cost reduction to assure the maintenance of the 
current level of mental health services without 
adversely affecting service quality. 

TYPE OF ORGANIZATION 

Membership on the CTF included three 
members of the community services board. In 
addition, the citizen boards of all mental health 
agencies were invited to appoint two board 
members to the CTF. One member of the board of 
Alternative House (a residential crisis interven- 
tion program for adolescents), one citizen 
representative from alcohol programs and one 
person representing mental retardation programs 
were also appointed. All of these members were 
volunteers from ci*'izen boards. 

Nine staff members were assigned on a 
part-time basis to assist the CTF. The staff 
arranged visits to programs, collected and 
analyzed data, and shared their professional 
points of view but were not involved in the 
decisionmaking process. 

The CTF organized itself into six committees 



For further information write Barbau*a Todd, Vice 
Chairman, Fairfax-Falls Church Conununity Services 
Board, 301 Maple Avenue West, 3rd Floor, Vienna, VA 
22180; or Barbara Todd, 312 Van Buren St., Falls 
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representing functional service delivery needs 
that cut across the three disability areas of 
mental health, mental retardation, and sub:tance 
abuse. The committees were: (1) outpatient/after 
care; (2) partial hospitalization/day treatment/ 
vocational; (3) inpatient/residential; (4) emer- 
gency services/rtcreation; (5) community 
services; and (6) consultation and education/ 
evaluation/transportation. 

EVALUATORS OR MONITORS 

All CTF members were selected because of 
their involvement as volunteer members of 
citizen advisory or governing boards. Every effort 
was made to include representat' ^ i from all 
disability areas and from a wide range of 
programs. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The CTF was formed to review the entire 
service delivery system and all levels of admin- 
istration of the Fairfax-Falls Church community 
services board and to make recommendations 
regarding improved service delivery, filling 
service gaps, and especially the operation of the 
system in a more cost-effective manner* Faced 
with the prospect of major budgetary restrictions, 
particularly those involving the stepdown of 
Federal funding in the mental health centers, it 
was necessary to explore all possible methods for 
reducing operational expenses. 

LEVEL OF PARTICIPATION 

The members of the CTF were directly involved 
in the following steps: establishing the direction 
and depth of the investigation, reviewing the 
programs and facilities, questioning the staff, 
requesting data or data analyses, soliciting 
information and ideas from staff and community, 
evaluating the information, developing recom- 
mendations, and writing the final report. 

TARGET OF EVALUATION 
OR MONITORING 

This investigation addressed the administrative 
und service delivery organization of the mental 
health, mental retardation, and substance abuse 
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programs sponsored by the Fairfax-Falls Church 
community services board. This community 
service board serves an area with a population of 
625,800 persons. Federal, State, and local 
governments share in funding these programs. 
Some programs are operated directly by the 
community services board. Other services are 
purchased from private nonprofit organizations. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The main thrust of this undertaking was to 
determine whether one or more alternative 
structures could be developed that could operate 
more effectively and economically while pre- 
serving the level and quality of service. Addi- 
tional concerns included the potential for 
improvement in quality and convenience of 
services and the elimination of any identified gaps 
in service. 



TECHNIQUES USED 

Each CI F committee visited all programs that 
provided services in the area of the committee's 
concern. Staff explained the service needs of the 
clients, the program staffing, administration, 
records, interrelationships with other programs 
and agencies, problems, and gaps. Reports 
describing the operation of the agency and its 
programs were made available to the committee. 
On the basis of the site visits and material 
reviewed, the six committees discussed and 
evaluated their findings and each developcsd a 
report to present to the entire CTF • In addition, 
44 presentations by program staff were given to 
the CTF during three evenings of hearings. 

FINDINGS OF EVALUATION 
OR MONITORING 

The CTF was in agreement that although most 
clients in the system were provided with appro- 
priate, high-quality service, several classifi- 
cations of clients were being either poorly served 
or not served at all. These included clients who: 

• Needed a variety of services within a 
disability area 

• Needed services in more than one disability 
area 

• Needed services from other social services 
agencies 

• Were geographically or socially isolated 
It was also the opinion of the CTF that the 



following conditions were detrimental to the 
development of a unified, effective, economical, 
administrative system: 

• The organization of service delivery differed 
among the three mental health centers. 

• There was no uniformity in administrative 
procedures such as billing, clinical records, 
purchasing, and personnel policies. 

• There was duplication of effort in areas such 
as training, public relations, and evaluation. 

• There were barriers to conrimunication and 
cooperation among agencies. 

• There was a need to strengthen systemwide 
planning and coordination. 

Th3 CTF estimated that, in addition to 
improved services to clients, the recommended 
restructuring of the system would result in a 
savings of approxime cely $i million over a 5-year 
period and would cffset the effect of the Federal 
grant stepdown until fiscal year 1986. 

RECOMMENDATIONS 

In regard to the organization of service delivery 
units, the CTF recommended the development of 
a comprehensive community services system that 
would include: 

• Integrating mental health, mental retardation 
and substance abuse capabilities at intake 

• Converting the mental health centers to 
community service centers more tightly 
networked with mental retardation and 
substance abuse delivery units 

• Highlighting the case management function 

• Establishing appropriate policies to ensure 
administrative i imitment to service 
integration 

• Assuring that advocates of mental retarda- 
tion and substance abuse services are 
represented on the center governing boards 

Recommendations to strengthen lateral 
cooperation of service delivery included: 

• Organizing each service center along 
common lines internally: youth and family 
services, adult services, community care 
programs, emergency services, and consul- 
tation and education 

• Providing systemwide client guidance at 
intake 
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• Sharing certain areawide specialized functions 

• Applying uniform personnel standards and pay 
policies 

• Developing uniform systemwide standards, 
forms, and procedures 

Service improvement and operational economy 
should result from selectively decentralizing some 
services and centralizing others: 

• Outpatient and "walk-in- counseling should 
be geographically decentralized to 
underserved populations in certain identified 
expanding areas. 

• There should be centrally administered 
systemwide planning, coordination, and 
resources for certain professional support 
functions: evaluation, quality assurance, 
training, consultation and education, and 
client indax. 

There should also be centralized planning, 
coordination, and resources for certain 
administrative functions: budget, trans- 
portation, personnel, purchasing, property 
management, public information, grants, and 
regulatory coordination. 

• Certain specialized services should be 
centralized at a single appropriate service 
delivery site. 

• When feasible, services should be colocated 
with other related human services. 

The structure of the community services board 
administration should be steamlined with clearly 
defined lines of authority to serv ce delivery units 
by: 

• Having the directors of the community 
service centers report directly to the 
executive director of the community services 
board 

• Providing coordination for contract agencies 
and for areawide programs in a division of 
special programs 

• Maintaining a high-level focus on substance 
abuse and mental retardation through 
appropriate administrative authority 

• Aggregating support resources for delivery 
units and for the executive director in two 
divisions: professional support functions and 
administrative support functions 

On issues where the CTF was in strong 
agreement, a single recommendation was 



presented. When the CTF was divided, the 
alternatives we**e offered in the report. 

STEPS TO ENSURE IMPLEMENTATION 

The 62-page final report of the CTF was 
published and distributed to all members of the 
community services board, the directors and 
boards of all affected agencies, interested client 
support organizations, and the governing bodies of 
the jurisdictions served. The report was also made 
available to all other interested persons. In order 
to facilitate the implementation of the recom- 
mendations, the CTF advised that a public hearing 
be held, that an implementation advisory council 
be established, and that an implementation 
timetable be developed. It was also recommended 
that wherever possible implementation be as 
prompt as possible in those areas where there 
appears to be strong agreement. 

The authority of the CTF was limited to 
undertaking an extensive study of the entire 
service delivery system and to developing 
recommendations for appropriace changes. The 
authority to effect any of the recommended 
changes rests with the community services board 
in those areas where policy changes are involved 
and with the executive director and his staff for 
matters involving implementation of policy. 

EXTENT OF IMPLEMENTATION 

Although this report was submitted in Decem- 
ber 1980, no formal action has yet been taken (as 
of July 1982) to approve the implementation of 
any of the 33 recommendations. A number of 
situations have contributed to this neglect: 

• At the time the report was presented, the 
community services board was functioning 
under an acting director and it was the sense 
of the board that implementation would only 
succeed under strong and enthusiastic 
encouragement from a permanent executive 
director. 

• Specific changes in operating procedures 
were being strongly encouraged by the county 
management (the fiscal agent for the 
comrnunity services board). This absorbed an 
inordinate amount of staff time which 
otherwise would have been devoted to 
implementation of CTF recommendations. 

• Many of the cost savings to be derived from 
the recommended changes in administrative 
services depended on the installation of an 
automated management information system 
for which specific recommendations are only 
now being completed. 

• The mental retardation support community is 
strongly opposed to the concept of an 
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integrated service system, mainly out of fear 
of losing the ••visibility'* it hei worked hard 
and long to establish* 

• Changes at the State level indicate that 
funding will continue to be distributed 
according to disability^ area — an obstacle to 
developing an integrated system. 

• Federal block funding of mental health 
center grants to the States has eliminated 
the requirements for a governing board for 
each mental health center, thus making it 
impossible to develop internally a strong, 
unified mental health system. 

Nevertheless, the CTF report has served as a 
set of guidelines for many changes that either 
have been or are about to be made in the service 
system. In the past year, the trend has been 
toward the development of a unified system of 
mental health, mental retardation, and substance 
abuse services with stronger ties to other human 
services. 

Greater concern is being shown for clients with 
special or multiple service needs and new 
procedures and service techniques are being 
developed for them. Case management is being 
extended to mentally ill and substance abusing 
clients who have been unserved or poorly served 
in the past. At the county adult detention center 
(i.e., local jail) services in all disability areas have 
been improved. 

At the three mental health centers, alcoholism 
services are available anc are being coordinated 
with mental health services. Progress is being 
made toward providing appropriate mental health 
services to mentally retarded clients. Contract 
agencies that originally provided vocational 
services to mentally retarded clients now have 
developed programs for the mentally ill. 

In the administrative structure of the com- 
munity services board, the establishment of the 
position of assistant director for services should 
ensure a unified approach and closer cooperation 
among all disability areas. All case management 
is now under a single director. The incorporation 
of the management of all residential program is 
currently in the planning stage. 

There has been an appreciable reduction in 
administrative personnel, especially in the mental 
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health centers. Some of this has been the result of 
combining small agencies under a single manage- 
ment or consolidating geographic locations of a 
single program. More progress in this area will be 
made when the management information system 
is in place. 

Considering the fact that the community 
services board has had to respond to a change in 
executive director and strong directives for 
changes from local. State and Federal funding 
sources, it would seem as though a reasonable 
number of the CTF's recommendations have been 
implemented. In a more stable situation, a formal 
approach to the implementation process might 
have been preferable. 

SPECIAL BARRIERS OR SUPPORTS 

The time available is always a barrier in 
undertaking any study because the need for the 
results is urgent before the task is initiated. 
Other than that, there were no problems. The 
group was enthusiastic and hard working. 
Everyone was willing to devote the time neces- 
Sc :y to complete the job. The staff was dedicated 
and supportive. All the agencies were cooperative. 

RESOURCES AND COSTS 

As closely as it is possible to estimate the time 
and effort expended in the project, the 15 citizen 
volunteers devoted approximately 4,000 hours to 
this effort and the 9 staff members spent about 
3,000 hours in organizational and support 
activities. Each citizen member drove approxi- 
mately 400 miles to attend an average of 25 
meetings- On the basis of 20^ per mile, this 
represents a contribution of some $1,000 in the 
interest of completing this task. 

ADDITIQNAL COMMENTS 

This was a long, laborious endeavor. Time was 
needed to establish mutual trust and willingness 
to negotiate differences. It required a commit- 
ment to the concept that this was a necessary and 
worthwhile activity that would benefit all clients 
and help all agencies. It was an appropriate and 
effective approach to a complex and difficult 
problem. This project could not have succeeded 
without the enthusiastic support of the staff. 
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34. A Board of Visitors Monitors Client Rights issues 

Kelly Moorse 

Montana Mental uisabilities Board of Visitors 



SUMMARY 

The Mental Disabilities Board of Visitors 
conducts annual reviews of Montana's community 
mental health centers, the State's three insti- 
tutions for the mentally ill, and its two institu- 
tions for the developmentally disabled. As part of 
each review, the board examines patient care and 
treatment, mecfication, recordkeeping, and 
consumer issues, i.e., client rights, grievances, 
etc* After each site visit, a report of the board's 
findings is sent to each facility, to the Governor, 
and to the director of the department of insti- 
tutions. In addition, board members together with 
staff respond to individual grievances regarding 
violation of rights, patient treatment, and other 
concerns. 

TYPE OF ORGANIZATION 

The Mental Disabilities Board of Visitors, 
established by the Montana Legislature in 1975, is 
charged by State law with reviewing patient care 
and treatment at Montana's community mental 
health centers, as well as its institutions for the 
mentally ill and the developmentally disabled. The 
board is administratively attached to the 
Governor's office and, therefore, is independent 
of any facility that it investigates. 

The board consists of five members who are 
appointed by the Governor. The board's role is to 
monitor all aspects of patient care and treatment, 
as outlined in the State's two principal statutes 
governing the mental disabilities system: The 
Mental Commitment and Treatment Act and The 
Developmental Disabilities Act. 

EVALUATORS OR MONITORS 

Of the five board members, two are consumers 
v^o represent the interests of the mentally ill and 
the developmentally disabled and three are 
professionals: a clinical psychologist, a registered 
nurse, and a lawyer. Consultants in pharmacy, 
medicine, psychiatry, gerentology, and special 
education also are available to the board on a 
contractual basis. 



For further information -write Kelly Moorse, Office of 
the Oovemor, Mental Disabilities Bd, of Visitors, 
Capitol Station, Helena, MT 59620, (406) 444-4765. 
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REASONS FOR THE EVALUATION 
OR MONITORING 

Montana law requires that each mental 
disabilities facility be visited at least annually. As 
an independent board of inquiry, the board of 
visitors is mandated to make annual site visits to 
those facilities within its purview. In addition, the 
board and its staff member must respond to 
individual grievances. 

LEVEL OF PARTICIPATION 

Board members are involved in every aspect of 
the facility reviews, including onsite visits to 
each facility. The board conducts an average of 
10 site visits per year, spending 2 to 3 days at 
each site together with the board's staff member 
and contracted consultants. As part of the site 
visits, board members review various treatment 
and medication plans, inspect the residential and 
treatment areas, and assist individual clients who 
may have grievances. After each site visit, the 
findings of the board members and consultants are 
submitted to the staff member who will collate 
the information into a report. In addition to the 
scheduled site visits, board members respond to 
hundreds of individual requests by patients, their 
families, and the legal system to review and 
investigate, among other things, care and 
treatment issues. 

TARGET OF EVALUATION OR MONITORING 

The board of visitors evaluates the 5 regional 
mental health centers, 1 or 2 of the satellite 
mental health centers that are located within a 
region (the board's budget does not permit review 
of all 35 satellite centers within the State) and 
the 5 State institutions. Three of the five 
institutions are mental health facilities: Warm 
Springs State Hospital, the largest state facilily; 
Galen's State Hospital; and the Center for the 
Aged. The remaining two facilities serve the 
developmentally disabled: Boulder River School 
and Hospital and Eastmont Human Services 
Center. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

During their site visits, board members conduct 
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the following activities: (1) ensure that treatment 
plans exi:^c and are being implemented and 
modified as needed for each patient; (2) ensure 
that medication reviews are conducted and that 
no medication is given as punishment or for staff 
convenience; (3) review the use of ail restraints, 
isolation and other extraordinary measures; 
(4) inspect the residential and treatment areas, 
recreational and dining facilities to ensure that 
they offer a therapeutic milieu; (5) review 
admissions and retention of patients to ensure 
that their legal rights have not been violated and 
that proper commitment procedures have been 
followed; and (6) assist patients in resolving any 
grievance regarding their commitment, alleged 
violation of rights or maltreatment at the 
facility. Other general concerns include record- 
keeping and staffing as they relate to overall 
patient care. 

A primary focus of the board of visitors is 
responding to individual grievances. Some 
examples of individual grievances that have been 
investigated by the board include the following: 

• A young woman at Warm Springs State 
Hospital was denied visitation, phone 
privileges, and home >'isits. Her family 
contacted the board of visitors, requesting 
that these rights be reinstated. 

• A teacher from the public school system 
referred a young developmentally disabled 
person to the board of visitors who was 
institutionalized at a mental health facility 
and wanted to be released. 

• A mental health center client contacted the 
board of visitors regarding a treatment plan 
that recommended inpatient thereapy at the 
state mental institution instead of outpatient 
services. An involuntary commitment hearing 
was requested by the mental health center at 
which time the board became involved. 

TECHNIQUES USED 

The board uses several techniques to assess 
patient care and treatment during a site visit: 
(1) interviewing patients and clinical and 
administrative staff; (2) observing patients and 
staff; (3) and conducting a random record review. 
The checklists that are used for the random file 
review are based on the standards established by 
Montana law (the medication checklist was 
developed for the board by a consultant). For the 
interviews and observations, an onsite review 
form is used that focuses on four major sections: 
(1) maintenance of records, (2) program and 
treatment plans, (3) training for resident case 
workers, and treatment of residents, i.e., 
rights, abuse issues, etc. 

In order to respond to individual grievances, the 
board has established certain procedures to follow 
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in those cases. Board members and the staff 
person respond immediately to the grievance and 
collect any relevant information regarding the 
complaint including existing documentation, 
previous resolutions (if any), and possible 
alternatives that are acceptable to the client. The 
information is then analyzed and, if necessary, 
additional data are collected from other resi- 
dents, facility staff or legal personnel. All of 
these resources are reviewed and possible 
solutions and alternatives are then discussed with 
the client. At that time, board members may 
advocate for certain changes on the unit where 
the client resides, request policy changes for the 
entire facility, request legal assistance, or work 
to change clinical and administrative staff 
attitudes. 

FINDINGS OF EVALUATION OR MONITORING 

Each site visit conducted by board members 
results in a list of facility accomplishments and 
deficiencies. Findings in each key issue area 
described earlier differ for each facility visited. 
With respect to client rights issues, the 1981 
board of visitors review of Warm Springs State 
Hospital expressed the following concerns: (1) a 
copy of a patient's rights was not promptly given 
to each person upon admission— in some cases it 
was 8 to 1 5 days before the patient received a 
copy; (2) the rights of patients were not always 
posted in each ward — a direct violation cf State 
statute; (3) procedures needed to be developed to 
reinforce the importance of a patient's rights 
throughout a person's stay in the facility; and 
W inservice training for staff regarding patients' 
rights, including the right to refuse treatment, 
needed to be implemented. 

Specific findings also emanate from the board's 
investigation and evaluation of individual 
grievances. For example, after investigating the 
denial of visitation and other privileges to a Warm 
Springs patient, the board found that there were 
written orders restricting phone calls, visitors, 
and home visits — such orders are required by law 
if any rights are restricted. In another situation 
concerning the institutionalization of a young 
developmentally disabled person who wanted to be 
released, the board met with the individual, 
completed the investigation, and determined that 
the individual was not seriously mentally ill as 
defined by the law (the staff of the facility 
concurred with the findings). Lastly, for the client 
who was requesting outpatient as opposed to 
inpatient services, the board met again with the 
client and her court-appointed attorney to discuss 
the impending court proceedings a.td alternatives 
available to the client. 

RECOMMENDATIONS 

Recommendations vary according to the 
deficiencies listed in each site visit report. With 
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respect to the findings of the board in its 1981 
report to Warm Springs State Hospital, the board 
recommenoed thet the facility post copies of 
patients' rights in each ward and provide them 
with a copy of their rights whenever necessary. 

Board members also make specific recom- 
mendations concerning individual grievance 
complaints. For example, for the young develop- 
mentally disabled individual who was inappro- 
priately institutionalized in a mental health 
facility, the board recommended that the 
individual be referred to the prerelease unit, that 
the necessary paperwork be filed to obtain 
supplemental income, and that a group home 
placement be sought for this individual. 

STEPS TO ENSURE IMPLEMENTATION 

Each site visit report is sent to the reviewed 
facility within 60 days of the review; the facility 
then has 30 days to comment on the findings and 
recommendations. At that point, the final report 
is sent to the Governor's office and to the 
director of the department of institutions. The 
board's staff member has overall responsibility 
for following up on any deficiency that is related 
to patient care. Monthly updates and changes in 
the reports are reported to the board members. 
Other deficiencies may be followed up with 
subsequent site visits by the board. In the event of 
a serious deficiency, the board members and staff 
meet with the director of the department of 
institutions or the Governor's staff to explore 
possible solutions. The board's findings are also 
highlighted in the media. 

With respect to the specific patient rights 
deficiencies discovered at Warm Springs, the 
board of visitors assumed the responsibility of 
refining and updating the existing procedures for 
informing patients of their rights. In addition, the 
board monitors the implementation of its 
recommendations regarding individual client 
grievances. 

EXTENT OF IMPLEMENTATION 

The facilities all respond differently to the 
board's fifidings. Some facilities are very resistant 
to change, while others use the reports to correct 
the noted deficiencies. In addition, many of the 
recommendations that are directed at removing 
deficiencies are contingent upon the appropriation 
of additional funds from the State legislature. In 
some instances, the facilities use the board's 
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reports in preparing their requests for funding 
from the legislature. 

With respect to the board's recommendations 
on client rights issues, four of the five mental 
health centers have implemented this policy 
within their regions. In addition, the board has 
sent a questionnaire to each State's protection 
and advocacy office in order to collect infor- 
mation on how States summarize patient rights: 
who informs clients of their rights; what re- 
sources are used in explaining rights to the 
developmentally disabled, the hearing or sight 
impaired; and if inservice training is provided, 
what models are used. 

Board members have been able to implement 
several of their recommendations regarding 
individual grievances. For example, two of the 
three recommendations concerning the inappro- 
priately institutionalized developmentally 
disabled individual have been implemented; 
however, the most critical recommendation, i.e.. 
the group home placement, has been delayed 
because of a lack of community placements. The 
board, however, was successful in reinstating 
phone calls, visitors, and home visits for the 
client who had these rights restricted. 

SPECIAL BARRIERS OR SUPPORTS 

The major barriers to the board of visitors are 
limited resources and staff to assist board 
members in carrying out their mandated 
responsibilities. 

RESOURCES AND COSTS 

Each board member receives a nominal fee 
($25) for each day of a site visit and funds to 
cover lodging, meals and travel expenses. 
Consultants are paid an honorarium ($100) plus 
expenses. In addition, the board employs one 
full-time staff member at $19,000 a year plus 
benefits to collate the board's information and to 
foliowup on individual problems and complaints 
throughout the State. 

ADDITIONAL COMMENTS 

The board of visitors is a legislatively mandat-'^d 
board of inquiry; however, its funding must be 
approved by the Montana Legislature every two 
years. Without this basic source of support, it 
would be difficult for board members to conduct 
site visits in all State facilities. 
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35. Consumers Evaluate Their Needs and Agency Programs 

Joyce G. Smith 

Hill House Mental Health Rehabilitation and Research, Inc., Cleveland, Ohio 



SUMMARY 

Since 1977, adult mental hea'th consumers 
representing Hill House, a psychosocial renabil- 
itation agency, have functioned as client advo- 
cates and designed several instruments for 
assessing client needs and evaluating client 
programs in three of the agency's programs: the 
social program, the vocational program, and the 
transitional housing program. Members of the 
Client-Oriented Program Evaluation (COPE) 
group submit the results of the surveys to the 
agency's staff and board of trustees in order to 
maK services, programs, policies, and procedures 
more responsive to client requirements. Some of 
the survey results and excerpts from the survey 
instruments have been disseminated to clients and 
professionals in the mental health field through 
public forums and professional publications. 

TYPE OF ORGANIZATION 

The Client-Oriented Program Evaluation 
(COPE) group consists of 15 volunteer clients or 
former clients of Hill House who meet weekly co 
discuss and design evaluation instruments. COPE 
works in. conjunction with the agency's research 
director under a grant from the Ohio Department 
of Mental Health. COPE is an autonomous group 
within Hill House that is providing evaluative 
information to the staff and board of trustees in 
order to assure that policies, procedures, and 
services meet client needs. 

EVALUATORS OR MONITORS 

All of COPE'S efforts are accomplished by its 
consumer members who volunteer to participate 
in evaluation activities; assistance is also 
provided by the Hill House research department. 
COPE members represent a cross section of 
consumers in terms of age, level of education, 
diagnosis, and socioeconomic background. 

REASONS FOR THE EVALUATION 
OR MONITORING 

In 1977, both the agency's research director and 
consumer members believed that clients should 
become involved In evaluating the efficacy of 
mental health services offered to them. More- 
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over, it was felt that clients would respond more 
affirmatively to an instrument developed by 
fellow clients who shared their experiences and 
respected their needs. As a result, COPE was 
formed to create a client-developed measured 
(COM) of client progress to replace a behaviorally 
oriented measure (COM was applied to all clients 
of Hill House). With the completion of this task 
and the prospect of continued funding, COPE 
decided to extend its evaluation activities to 
create a needs assessment questionnaire and 
several instruments to evaluate each of the 
agency's three components. The fundamental 
question underlying the involvement of clients in 
a needs assessment and program evaluation was 
whether clients, as unsophisticated researchers, 
could clearly determine client progress and client 
satisfaction with agency services and make 
realistic recommendations for changes in the 
agency. 

LEVEL OF PARTICIPATION 

COPE members have been involved in every 
stage of the evaluation project undertaken at Hill 
House. Beginning with the creation of the 
client-developed measure (COM), clients wrote, 
edited, and revised the questions that became the 
final version. Client evaluators also studied the 
pretest results and were able to tailor the lengthy 
pretest version of the COM down to its current 
size. 

COPE members have worked closely with the 
agency's research staff, client coordinators, and 
other agency professionals in developing the 
evaluation program at Hill House. As COPE's 
involvement in evaluation expanded and pro- 
gressed through various stages, the number of 
agency clients represented in the process also 
grew. The initial success of COPE's involvement 
in the COM prompted the group to approach other 
evaluation topics, using the feelings of clients as 
the foundation for further research. 

TARGET OF EVALUATION 
OR MONITORING 

The COPE group has focused its efforts on 
assessing client needs and evaluating client 
programs provided by tHill House Mental Health 
Rehabilitation and Research. Hill House is located 
in the heart of the educational, cultural, and 
social service area on the east side of Cleveland 
(University Circle); it also operates a branch on 
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the west side of the city. For over 20 years, the 
agency has served former patients from Clove- 
und*s public and private mental health hospitals 
as well as persons with mental or emotional 
problems who have never oeen hospitalized. The 
program is administered oy an executive director 
and governed by a 30-member board of trustees, 
three of whom are consumers or former client 
members of the agency. 

A total of 33 staff, 19 of whom are profes- 
sionals, work with approximately 592 clients per 
year. The principal component of Hill House is its 
social rehabilitation program. This program 
consists of group sessions led by social workers, 
and sometimes by clients, to explore subjects 
proposed by clients and staff. These sessions are 
supplemented by individual client-worker 
contacts and by client-initiated social activities, 
such as meeting informally over lunch in the 
agency's dining facility* Two other Hill House 
programs offer services to clients: (1) the 
vocational program serves 200 clients a year and 
provides training in clerical, food service, 
Tiaintenance skills, and in job-seeking skills; and 
(2) the cooperative transitional housing program 
serves 24 clients a year. 



PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The first research focus undertaken by COPE 
members was to design a self -assessment 
instrument that totally reflected the client's 
perspective on mental health and treatment 
progress. In developing CDM, COPE members 
discovered that while the clinician's focus was 
primarily on behavior as an indicator of improve- 
ment or deterioration, clients responded much 
more readily and positively to an orientation 
around feelings and needs. As a result of its 
success with the CDM, COPE decided to apply the 
"feelings" orientation to the other Hill House 
program, ^gain, using clients as respondents. In 
addition, COPE's funding source requested that it 
develop a needs assessment instrument for Hill 
House clients. 



TE CHNIQUES USED 

COPE'S members developed and validated 
several questionnaires designed to measure the 
feelings and needs of clients and to evaluate the 
three programs at Hill House — the social 
program, the vocational program, and the co-op 
housing program. The questions for the 
instruments wore based on COPE members' life 
experiences and suggestions from other Hill House 
clients. Two of the three instruments used were 
structured questionnaires. The third instrument 
was an open ended survey that was administered 
to the clients in the cooperative. 
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FINDINGS OF EVALUATION OR MONITORING 

The client-developed measure (CDM) ques- 
tionnaire was the first client instrument to be 
administered and completely validated by COPE 
at Hill House. This evaluation demonstrated both 
to the agency and to COPE that clients can design 
a valid and reliable instrument. The results of the 
needs assessment and program evaluation 
instruments, though promising, are not final. The 
status of each instrument is as follows: the needs 
assessment questionnaire has been administered a 
second time; the social program questionnaire has 
been administered to a larger sample of clients; 
and the vocational program instrument is being 
revised and will be administered a second time. 
The transitional housing interviews are complete. 

Preliminary findings indicate that both the 
needs assessment and program evaluation 
instruments do measure client satisfaction. COPE 
members believe that the initial evaluation 
results suggest possible changes in Hill House 
programs; however, no formal conclusions will be 
drawn until the survey results are completed. 

RECOMMENDATIONS 

Since the initial findings of the needs assess- 
ment and program evaluations are still being 
reviewed by COPE members, no formal recom- 
mendations have been developed concerning these 
survey results. The successful involvement of 
consumers in developing the CDM, however, led 
to certain overall recommendations. For example, 
it was recommended that the CDM group expand 
its activ :ies to include program evaluation and 
needs assessment, and to increase consumer 
representation in these evaluation functions. 
Furthermore, it was recommended that clients 
increasingly represented on the Hill House Board 
of Trustees and in other agency programs. Both of 
these recommendations have been implemented 
successfully. 

STEPS TO ENSURE IMPLEMENTATION 

The major implementation efforts to date have 
focused on the results of client involvement in the 
CDM. During the past five years, a strong 
cooperative relationship has developed between 
the COPE group and Hill House staff which, in 
turn, had led to increased visibility of consumers 
in agency functions. COPE has further enhanced 
its credibility by disseminating its process and 
news of its progress within the State of Ohio and 
throughout the nation by making presentations at 
various mental health conferences, meetings, and 
workshops. In addition, both COPE and Hill House 
staff have prepared articles for several mental 
health publications that describe the consumer- 
based evaluation system. 

In order to implement the final results of the 
program evaluation and needs assessment, COPE 
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organized a daytime auxiliary of Hill House 
clients (COPE II) who are responsible for de^'sl- 
oplng groups and activities that are Indicated by 
the questionnaire responses. 

EXTENT OF IMPLEMENTATION 

As ev lu J by the existing level of parti- 
cipation of the COPE group, the recommendatlors • 
concerning Increased client Involvement In 
evaluation activities have been Implemented. 
Moreover, there Is Increased client representation 
on the Hill House board of trustees, and two paid 
COPE coordinators, both former clients, were 
recently hired to focus exclusively on evaluation 
activities. 

Despite the lack of formal recommendations 
emanating from the program evaluations and 
needs assessment, several changes have already 
been made In agency operations because of 
COPE'S established credibility In earlier activ- 
ities. For example, the original staff -developed 
assessment Instrument was replaced by COM; the 
needs assessment questionnaire Is now used as an 
Intake Instrument for all new clients; and the 
transitional housing residence limitation has been 
extended, when appropriate, from six to nine 
nnonths. In addition, COPE II and the COPE 
coordinators are making dally changes at Hill 
House that reflect new programs and activities 
such as Reaching Out, a group that focuses on 
group skills, parenting (for those Hill House 
nnembers who are parents), and food and clothing 
collection. Weekend activities planned by clients 
such as picnics, potluck suppers and trips to 
community areas of interest also have Increased 
and are now a growing part of Hill House. Lastly, 
because many clients wanted Hill House to 
attract outside speakers to address legal rights, a 
monthly seminar was Initiated that has been very 
well attended by consumers. 

SPECIAL BARRIERS OR SUPPORTS 

Changes In agency research directors have been 
a hindrance to COPE's work. On the other hand. 



without the technical support of the research 
directors, COPE would not have been able to 
achieve what It did. 

The need to educate new members on the 
complexities of COPE's research has, from time 
to time, temporarily slowed the group's momen- 
tum; however, COPE's "hair-down" rap sessions 
about Individual mental Illness experiences not 
only keep the "feeling" base olive for the research 
but also apppear to release the tensions of new 
members and to motivate them to participate. 

Hiring two former Hill House clients as COPE 
coordinators has been a tremendous support. 
These Individuals are responsible for admin- 
istering the questionnaires, organizing and leading 
meetings with staff and clients, soliciting 
feedback, and exploring Implications for im- 
proving the quality of services. 

RESOURCES AND COSTS 

Each COPE nnember donates at least 100 hours 
annually to the organization. Some members, 
especially the chairperson and vice chairperson, 
give more than 100 hours. Other staff time 
allocated to COPE activities Is as follows: 
research director— 25 percent; two COPE 
coordinators— each 100 percent; project sec- 
retary— 60 percent; statistician- -10 percent; and 
data processor — 30 percent. Some monies are also 
allocated for reproduction, keypunch rental, 
clerical supplies, tape cassettes (all meetings are 
recorded), travel expenses, and computer time. 
Financial support for all of COPE's projects has 
been provided by the office of program evaluation 
and research of the Ohio Department of Mental 
Health and the Cleveland Foundation. 

ADDITIONAL COMMENTS 

The clients at Hill House are enthusiastic about 
COPE'S work and they identify with, and support, 
the organization. Furthermore, these clients are 
anxious to share their experiences in an evalua- 
tion context in order to expand and strengthen 
COPE'S scope of work. 



36. Consumers Evaluate Community Residential Programs 

Cheryl Fanning 

Community Support Project, Arizona Department of Health 



SUMMARY 

Three advocacy groups in the State of Arizona 
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participated In a review of a community mental 
health residential treatment system for chron- 
ically mentally 111 (CMI) Individuals that was 
initiated as a result of legislation enacted in April 
1980. The legislation (ARS 36-550) provided funds 
for planning end Implementing a statewide 
community residential treatment system. The 
Arizona Department of Health's bureau of 
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community services was assigned the task of 
developing and monitoring contracts for these 
community residential and day services. Mental 
health agencies in five regions of the State were 
awarded funds to provide a continuum of resi- 
dential services. In order to solicit citizen views 
regarding these programs, advocacy groups were 
requested to visit the programs and submit their 
comments to the Department. 

TYPE OF ORGANIZATION 

Three of the five advocacy groups asked to 
participate in the review of the residential and 
day programs agreed to complete the evaluation. 
They included: Education, Advocacy, Support and 
Experience (EASE) from Tucson; Mental Health 
Advocates Coalition; and the human rights 
committee of the Arizona State Hospital. All of 
these groups are composed of consumers, family 
members, or other advocates for improved mental 
health services in Arizona. 

EVALUATORS OR MONITORS 

Reviews of the residential programs in seven 
Arizona cities were conducted by: a woman with a 
mental illness and her husband from EASE; the 
father of a CMI person from Mental Health 
Advocates Coalitior*; and three persons from the 
human rights committee- -a nurse, a 
rehabilitation coordinator, and a patient advocate 
from the Stat3 hospital. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The Arizona Department of Health's bureau of 
community services staff are responsible for 
monitoring contracts under ARS 36-550. The 
assistance of advocacy croups was solicited in 
order to evaluate the quality of residential 
services and to develop funding recommendations 
for the upcoming year. 

LEVEL OF PARTICIPATION 

Each advocacy group visited the programs at 
different times in order to conduct independent 
reviews. Reviewers scheduled the site visits with 
a contact person from each agency. 

TARGET OF EVALUATION OR MONIT C^^^iG 

The focus of the evaluation were the five 
primary contractors (and additional agencies with 
which they subcontract), that are funded under 
ARS 36-550. Northern Arizona Community 
Guidance Center, located in Flagstaff, is 
responsible for service delivery across four 
counties. The counties range in population from 
60,000 to 80,000; one county is larger in area than 
the State of Rhode Island. Several small day- 
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treatment and residential programs provide 
services in these counties. Behavioral Health 
Agency of Central Arizona is located in Pinal 
County, which has a population of approximately 
1,000,000. La Frontera Center, located in Tucson, 
funds several residential programs that were 
evaluated by advocacy groups. Community 
Behavioral Sciences and its two residential 
services subcontractors are located in Maricopa 
County— the most populated county in the State 
with 2,000,000 people. Maricopa County depart- 
ment of health services is also located in 
Maricopa County, as is Arizona State Hospital. A 
large percentage of deinstitutionalized chron- 
ically mentally ill persons reside in this county. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

This evaluation focused on three major areas: 
(1) physical aspects of the residence including 
space and cleanliness; attractiveness and 
comfortableness of furnishings; and presence of 
plants, wall hangings, personal belongings, and 
other items that contribute to a homelike 
atmosphere; (2) activities in which the residents 
were engaged such as making crafts, participating 
in socialization groups, reading and discussing 
current events in the newspaper- and (3) obser- 
vations of staff interactions wich patients. 

In addition, evaluators were asked to compare 
the existing level and quality of services to the 
previous year when the services were first 
initiated and to provide their overall impressions 
of the program. Most Importantly, evaluators had 
to determine if they could recommend a facility 
for a loved one who was mentally ill. Other 
recommendations were also solicited. 

TECHNIQUES USED 

Reviewers made site visits to each facility in 
the State using an open ended fjuestionnaire to 
record their observations and recommendations. 
The questionnaire, developed by the community 
support staff, focused on five key areas: (1) a 
description of the physical structure; (2) a 
description of residents' activities; (3) an 
assessment of staff and resident interaction; (^) a 
comparative analysis with last year's evaluation; 
and (5) an overall innpression of the program. 

FINDINGS OF EVALUATION OR MONITORING 

Although the specific findings of the evaluators 
varied for each facility, overall, their if.ipressions 
of the facilities were positive. As several 
reviewers commented, **I would feel comfortable 
in placing a person I care about in this facility.** 
Many comments addressed the physical environ- 
ment of the residence and included such de- 
scriptions as **neat," "clean,** **a healthy environ- 
ment,** or "the area was clean, roomy, well- 
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lighted, cooling and heating excellent.** Several 
comments focused on the need for better and 
sturdier furniture. All of the reviewers com- 
mented on the location of the program. Several 
individuals indicated that the program would be 
better located away from an inner -city, 
congested area. One reviewer noted that the 
location of the residence away from the central 
city area may have contributed to the positive 
attitudes of the residents. Another reviewer, 
however, observed that the residence was too far 
way from the nearest town and the crisis center. 

Other comments focused on the programing and 
administration of the various Tacilities. Several 
evaluators expressed their reservations about 
housing CMI's with alcoholics and drug abusers in 
the same facility. Other reviewers were im- 
pressed with the emphasis on promoting inde- 
pendence in some of the facilities. One reviewer 
noted *'^>»*' :. uay-treatment program also 
emphasized mainstreaming and promoting 
independence for Its clients. Another evaluator, 
however, had an overall negative impression of a 
facility because of the physical environment— 
i.e., the facility was overcrowded and had a 
dreary atmosphere. 

RECOMMENDATIONS 

Advocates made many recommendations 
concerning the findings described above. The 
recommendations were submitted to the Arizona 
Department of Health's bureau of community 
services staff who, in turn, shared them with the 
directors of the five major contracting agencies. 
The recommendations focused on several key 
concerns: physical plant considerations, location 
of the facilities, client groupings, and possible 
expansion of the programs. Some of the specific 
recommendations included: moving one residence 
closer to the nearest town in order to access the 
crisis center; providing heavier and sturdier 
furniture in the living areas and establishing more 
units at this particular site; t;xamining the 
possibility of separate living quarters for the 
chronically ill (CMI), alcoholics and drug abusers; 
developing more transitional housing; relocating 
an entire program (17 persons) to a better 
physical environment; and expanding and con- 
tinuing day programs and other residential 
programs for those on waiting lists. One reviewer 
noted: "There is a need "or ways to be found to 
move those [CMIs] In this program [who are ready] 
out into the community at large." 

STEPS TO ENSURE IMPLEMENTATION 

Department of health staff use tne evaluative 



comments prepared by the mental health 
advocates in allocating funds to providers. As 
such, providers have a fiscal incentive in up- 
grading their facilities. 

EXTENT OF IMPLEMENTATION 

With respect to the most recent set of evalua- 
tions conducted, certain agencies with identified 
deficiencies in the!" physical plants have already 
taken some steps ti locate other residential 
facilities. One of the facilities that received a 
poor rating last year — the first year the eval- 
uation was conducted by the advocates — receiv«)d 
a fairly high evaluation this year because of the 
marked improvement in its operation. 

liPECIAL BARRIERS OR SUPPORTS 

One of the advocates made a csnerally poor 
impression on the staff and the clients in the 
facilities, including being late for appointments 
and attempting to do "therapy" with a resident. 
This led to several written complaints to the 
department of health about this advocate's 
demeanor. In addition, the geographic distances 
involved and the extensive amount of time 
necessary for this type of program review limited 
participation in the evaluations. Only a few 
members from three advocacy groups actually 
made site visits and prepared written reports. 

Staff and residents of most of the facilities 
welcomed the evaluators and wanted to "show 
off" their programs to outsiders. 

RESOURCES AND COSTS 

Approximately $1,200 in community support 
project funds was used to reimburse the 3valua- 
tors for travel expenses. Other than preparing the 
evaluation format and drafting a reimbursement 
form, community support project staff spent very 
little time on the evaluation. The advocates spent 
six full days visiting all of the programs, and 
additional time was required in order to schedule 
the site visits. 

ADDITIONAL COMMENTS 

In general, the evaluators from EASE were 
pleased with the evaluation form but suggested 
that, for future evaluations, a checklist with a 
continuum of choices, i.e., "good," "better," 
"best," and other format changes might be more 
helpful. Once the visit was completed this 
checklist could then be used to write a "tran- 
script" report. 



126 



37. A Volunteer Evaluates the Delivery of Mental Health 
Services to Nursing Home Residents 

Margaret S. Munford 

In-Home Services Committee of the Washington County Council on Aging 



SUMMARY 

The In-Home Services Committee of the 
Washington County Council on Aging conducted 
an evaluation of the Tualatin Valley Mental 
Health Clinic's ability to address the problems of 
elderly residents in nursing homes. The evaluation 
results werj compiled in a final report that was 
included in the area agency on aging plan 
prepared by the Washington County Council on 
Aging. 

TYPE OF ORGANIZATION 

The In-Home Services Committee, a subcom- 
mittee of the Washington County Council on 
Aging, advises the council v)n a variety of services 
provided under the Older Americans Act, 
including mental health services. In Washington 
County, these services are provided under a 
contract with the Tualatin Valley Mental Health 
Clinic. 

EVALUATORS OR MONITORS 

This study was conducted by a member of the 
In-Home Services Committee, who is a retired 
school teacher with a master's degree in educa- 
tion and a strong background in sociology and 
psychology. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The In-Home Services Committee is responsible 
for evaluating the Tualatin Valley Mental Health 
Clinic's cont.dct *vith Lhe Ccjncil on Aging. Each 
year an evaluation is conducted to determine if 
the clinic's goals have been met. 

LEVEL OF PARTICIPATION 

In designing and implementing the study, the 
evaluator worked closely with the Council on 
Aging, the director of the Tualatin Valley Mental 
Health Clinic, the supervisors of the participating 
nursing homes and, if appropriate, their social 
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directors and the members of the In-Home 
Services Committee. The evaluator collected and 
analyzed and wrote the final report. 

TARGET OF EVALUATION OR MONITORING 

This evaluation focused on a single dimension of 
a single agency, that is, the delivery of mental 
health services to senior citizens in nursing homes 
as conducted under contract by the Tualatin 
Valley Mental Health Clinic. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The isp jes surround ng the delivery of mental 
health services to elderly persons confined to care 
centers were the focus of this evaluation. Certain 
questions pertaining to quality of care and 
distribution of services were addressed in the 
evaluation. These included: (1) a breakdown of 
clients according to diagnosis, age and sex; (2) a 
distribution of clients according to city of 
residence; and (3) the number of clients who live 
in nursing ncmes (each home was identified) or in 
their own homes. Client satisfaction with the 
delivery of mental health services was also 
assessed. Costs, however, were not included in the 
evaluation; they were reviewed and evaluated by 
the finance department of the area agency on 
aging. 

TECHNIQUES USED 

First, a list of questions concerning the quality 
and distribution of services to elderly persons 
participating in the mental health program was 
submitted to the clinic staff for completion. 
Second, consultations were held with the super- 
visors of the nursing homes and the director of 
the mental health clinic. Third, the evaluator 
attended a group therapy class to observe and 
evaluate the session. 

FINDINGS OF EVALUATION OR MONITORING 

Discussions with the nursing home managers 
involved in the mental health clinic program 
revealed a high level of enthusiasm for the 
program and the staff conducting it. In their 
opinion, the mental state of the nursing home 
patients had improved because of the program. 
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The therapy session observed by the evaluator was 
led by a staff person who was knowledgeable and 
well prepared. The session provided mental 
stimulation for those attending and a channel for 
relieving tensions through active participation in 
the meeting. One woman summed up her feelings 
about the program by noting that she could do 
nothing about her physical condition but that 
these sessions gave her something to think about 
when she was alone or in bed at night. 

The data abo'jt quality ^f care and distribution 
of services collected by the evaluator also 
revealed certain findings. In general, the data 
showed that the age distribution among the 
clients served by the program was very similar to 
the age distribution among the general population, 
but there was a sharp increase in the number of 
females aged 85 and over receiving services. 
Moreover, of the U9 clients with identified 
diagnoses, 63 percent (94 clients) had either 
organic mental disorders or conditions not 
attributable to a mental disorder that were the 
focus 0^ dttention or treatment (e.g., adjusting to 
a major illness or change of resioence). lo 
addition, of the 154 persons served by the 
program, 128 were living in nursing homes— an 
expected outcome since the program is targeted 
to clients of nursing homes. Overall, the evalua- 
tion results demonstrated that the Tualatin Valley 
Mental Health Clinic fulfilled its contractual 
agreement with the Council on Aging in every 
aspect. 

RECOMMENDATIONS 

The evaluator recommended that the clinic's 
contract be renev^ed for another year, that the 
Council on Aging increcis^ its service contract 
with the clinic, wr.cn finances permit, and that 
this report be used as a guideline for future 
evaluations. 

STEPS TO ENSURE IMPLEMENTATION 



members of the In-Home Services Committee. In 
its annual budget review, the Committee recom- 
mended to the Council on Aging that the Tualatin 
Valley Mental Health Clinic's contract be 
renewed. 

EXTENT OF IMPLEMENTATION 

The Tualatin Valley Mental Health Clinic's 
contract was renewed, and the results of the 
evaluation were incorporated into the area agency 
on aging plan prepared by the Washington Council 
on Aging. 

SPECIAL BARRIERS OR SUPPORTS 

The evaluator had the support of the directors 
of the Council on Aging and cooperation from the 
contractor, clients, ard supervisors of the nursing 
homes. 

RESOURCES AND COSTS 

There was no cost to the Council on Aging for 
this evaluation. The evaluator donated approxi- 
mately 20 to 25 hours to complete the study. 

ADDITIONAL COMMENTS 

The evaluation of the clinic's program on aging 
was a pilot project. As such, the final report was 
designed to provide all of the relevant facts about 
the program with the least amount of verbiage 
and to be useful to the recipient audiences. In 
developing the evaluation format, the c'lidelines 
for evaluation that are presented in the ^ontract 
Management Manual for Aging Programs (Bob 
Curry and Associates) were used and could be 
adapted for evaluations of other small social 
service contracts. 



The recommendati ns were presented to the 

38. Parents Monitor Group Homes to Ensure Quality of Services 

Ruth M. Taylor 

Parent Monitoring Committee, Associatiop fjr the Macomb-Oakland Regional Center 



SUMMARY 

In 1980, the Association of Macomb-Oakland 
Regional Center (AMORC), a parent advocacy 



For f linker information write Ruth M. Taylor, 
chairperson, Monitoring Committee, Association for 
Macomb-Oakland Regional Center, P.O. Box 471, New 
Haven, Michigan 48048, (313) 749-3038. 
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organization, formed a monitoring committee to 
respond to the concerns of parents and other 
individuals who have relatives living in community 
group homes tnal arc <5erviced through the center. 
The committee has monitored over 90 group 
homes to date and all of the concerns, strengths 
and weaknesses of the homes are discussed among 
committee members and then reported directly to 
the director of the Macomb-Oakland Regional 
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Center (MORC). An annual report is also pre- 
sented to the administrators of the group homes. 

TYPE OF ORGANIZATION 

The parent monitoring committee is a special 
unit of the Association for the Macomb-Oakland 
Regional Center — an advocacy group that acts on 
behalf of the clients receiving services either in 
Macomb-Oakland or in community programs. The 
role of the parent monitoring committee is to 
monitor the agency's group homes to ensure that 
the persons living in those homes are receiving 
the best services possible in an environment that 
enhances and motivates individual growth. 

EVALUATORS OR MONITORS 

The parent monitoring committee consists of 
approximately 30 volunteers who have develop- 
mentally disabled relatives living either in the 
natural home or in a community group home. 
Within the monitoring system itself, there is a 
core committee of seven persons who, together 
with other monitors, visit the community homes. 

REASONS FOR THE EVALUATION 
OR MONITORING 

In early 1980, AMORC's board decided to form 
a committee to address the need for monitoring 
the community system. The members of AMORC 
believed that such 3 monitoring system would help 
to alleviate parental concerns regarding the 
continuation of quality community homes once 
parents gre no longer living. In addition, the 
parent monitoring committee would complement 
two other monitoring efforts already used by 
MORC: (1) f/uality-of-Life Review Teams that 
are composed of nonprogram MORC staff who 
visit the homes monthly; and (2) case managers 
who visit the homes on a weekly basis. 

After many organizational meetings, the 
monitoring committee was ready to begin visiting 
group homes; however, it was not until June, 
1980, that the actual monitoring began. After 
conducting unannounced visits to more than 70 
group homes, discussing concerns, providing 
suggestions, and winning the praises of the 
director of MORC, committee members decided 
that the monitoring system was an asset and 
shniild be made a permanent part of the agency's 
function. 

LEVEL OF PARTICIPATION 

In conducting group home visits, a monitoring 
committee core person is always accompanied by 
another monitor. An evaluation form is filled out 
by the mor'tors after each visit, and the home 
reports are evaluated by the core committee 
during their monthly meetings. A core person then 
attends a monthly meeting with the director of 
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MORC and the case management supervisors to 
debrief t.iem on the results of the mcnitoring 
visits. 

TARGET OF ZVALUATION OR MONITORING 

The parent monitoring committee focuses cn 
the quality of services that are provided to the 
clients living in MORC community group homes. 
MORC contracts with nonprofit corporations to 
administer and provide services to clients living in 
the community. The ages and number of persons 
living in the homes vary; typically, there are 4 to 
12 persons residing in each group home. MORC 
staff provide the necessary support services to 
the clients including psychology, psychiatry, 
social work, nursing, and dietary. In addition, a 
MORC human rights committee monitors behavior 
management programs in the community. 
Community resources such as dentists, doctors, 
and speech pathologists are also used by group 
home residents. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The major areas that the monitors review 
during their home visits include: general quality 
of life and environmental issues, health, nutrition, 
and client rights. Among the issues of graatest 
concern to the committee are nucrition, inside 
and outside appearance of the home, client-staff 
ratios and compatibility, barrier-free 
accessibility for the multihandicapped and the 
proper storage of medications. Program 
implementation and client participation in 
community social and support systems are also of 
concern to the committee. 

TECHNIQUES USED 

After organizing the AMORC monitoring 
comrittee and core committee, the program was 
presented to the director o^ Macomb-Oakland 
Regional Center, who sanctioned it and notified 
the group home administrators of its purpose. 
Guidelines and monitoring procedures were 
established by the agency and core committee. 
For example, the core committee neets monthly 
and, during that time, each person on the 
committee is responsible for monitoring a certain 
number of homes with another monitor. Each core 
person accepts responsibility for monitoring the 
homes that are within their own geographic 
location. Checklists for the home visits were 
developed based on information from the 
following sources: input from core committee 
members; checklists used jy other agencies; and 
parental concerns. Finally, in order to enter the 
group homes, each core person was issued an 
identification card. 

Since June 1, 1980, approximately 4,000 
unannounced visits have been made to 1 14 group 
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homes under contract to MORC. Unannounced 
visits are used in order that the monitors can view 
the conditions in and around the homes in a candid 
and open manner. Monitors visit the homes in 
pairs; before entering each home, they make note 
of the security measures and the upkeep of the 
grounds. Further, each core person staggers the 
schedule for the home visits so that each home is 
observed under different circumstances. 

FINDINGS OF EVALUATION 
OR MONITORING 

Overall, the committee members have found 
that the group homes are functioning like normal 
home settings. The training courses required 
before any staff member can work in a group 
home are excellent (two core persons from 
AMORC monitored and critiqued this 3~week 
orientation course). Another significant finding 
was that the direct care staff turnover was 
evident in many of the group homes; however, the 
home managers have been successful in keeping 
client-staff ratios stable. 

RECOMMENDATIONS 

Based on the monitoring visits completed during 
1980-81, the core committee recommended that 
direct care staff for group homes receive a salary 
increase and that a graduated pay scale be used as 
an incentive to obtain increased wages and to 
retain staff. The committee also suggested that 
the pay scale be based on length of employment, 
experience, and level of responsibility. If imple- 
mented, these two recommendations would help 
to ensure a stable environment for clients living 
in community hor.'^es. 

STEPS TO ENSURE IMPLEMENTATION 

After reviewing the home visit overviews 
written by all core committee monitors, the 
chairperson of the committee compiled an annual 
report for 1981 that was sent to MORC and 
shared with the group home administrators. The 
monitors* concerns regarding the management and 
staff turnover in a number of the homes as well as 
their recommendations to resolve these problems 
were discussed with the administrators. A copy of 
the annual report was also forwarded to the 
director of the Michigan Department of Mental 
Health with a note encouraging his department, 
based on the monitoring committee's concerns, to 
seek increased funding in this area. 

If a monitoring report should highlight a par- 
ticular concern in a group home, the MORC 
reporter has 10 days in which to take corrective 
action. If the committee believes that appropriate 
corrective actions are not being taken, it can then 
contact the office of recipient rights, Michigan 



Protective and Advocacy Services, or other 
appropriate agencies. 

EXTENT OF IMPLEMENTATION 

At a recent meeting of State agency staff that 
included a member of the board of group home 
administrators, it was implied that home mana- 
gers would receive an increase in their operating 
budget for FY 1982-83. As a result of this in- 
crease, a "trickle down" effect would be felt by 
the direct care staff. In addition to this effort, a 
much greater increase in wages is needed as an 
incentive for staff longevity; however, funding 
constraints have inhibited implementation of this 
particular recommendation. 

SPECIAL BARRIERS OR SUPPORTS 

This new and unique way of monitoring group 
homes was needed, wanted, and eagerly accepted 
by Macomb-Oakland Regional Center staff. It was 
so well accepted that the format has been re- 
quested by numerous associations for retarded 
citizens and other organizations and agencies 
worldwide. In addition, MORC released a special 
edition of their newsletter. Transition, that was 
entirely devoted to the AMORC monitoring 
committee and the agency's acceptance of the 
program. 

A major concern of the committee is the recent 
cutback in funds allocated to regional centers, 
such as MORC, for gro jp homes. Michigan is 
currently in a severe recession, and its tax base is 
shri.^king because of a 14-percent-plus unemploy- 
ment rate. Unless the State's economic circum- 
stances improve, adequate funding for community 
homes will be in jeopardy. 

RESOURCES AND COSTS 

Since 1980, the AMORC monitoring core mem- 
bers have donated countless hours to the mon- 
itoring program. There has been, however, no 
estimate developed of the total number of hours 
spent by the committee members. MORC has 
provided an office, phone, mail box, mileage, 
materials, and use of a photocopy machine to the 
committee. 

ADDITIONAL COMMENTS 

The committee members are proud of their 
varying experiences and their ability to "keep an 
eye" on all group home residents. These activities 
have brought them together in a common bond. 
Parents can be candid and open in alerting core 
committee members about problems in the group 
homes; they also can be assured that their con- 
cerns will be investigated by the committee and 
monitored until necessary corrective action is 
taken. 
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39. Citizens Participate in Planning Services for Older Adults 

Ralph W. Accardi 

Older Adult Services Advisory Committee of the Marin County Mental 

Health Advisory Board 



SUMMARY 

Each year, Marin County Community Mental 
Health (CMHS) establishes citizen task forces as 
part of its planning process. Between 75 and 100 
persons participate in these task forces and 
develop recommendations aimed at initiating, 
expanding, or improving community mental health 
services. The recommendations are usee by CMHS 
staff to develop the final mental health service 
plans. Standing advisory committee also may 
submit recommendations, or they may comment 
on task force recommendations. All recom- 
mendations are based on assessment of needs and 
evaluation of progress in implementing the 
previous year's plan. The focus of one task force 
is services to older adults. 

TYPE OF ORGANIZATION 

Marin County Community Mental Health Serv- 
ices (CMHS) is a State-mandated, locally con- 
trolled agency within the Marin County Depart- 
ment of Health and Human Services. CMHS is 
responsible for developing and coordinating a 
comprehensive system of programs to meet the 
county's mental health needs. These programs 
address the problems of acute and chronic mental 
disorders, life crises, developmental disabilities, 
and drug abuse. Services may be provided directly 
by CMHS, purchased from community-based 
agencies, or provided by the private sector. 

The mental health advisoiy board (MHAB) is a 
State-mandated advisory group that makes 
recommendations to the mental health director 
and to the board of supervisors regarding the local 
CMHS program. The older adult services advisory 
committee (OASAC) is one of five standing 
committees that advise the MHAB and/or CMHS 
staff about services to special populations. In 
addition to monitoring and advocating on a con- 
tiniiing basis for mental health services directed 
to older adults, OASAC members also serve on 
planning task forces in order to evaluate existing 
services to older adults and to recommend new 
services as needed. OASAC members service on 
several planning task forces, such as one on 
mental health promotion and one on State hospital 



For further information write Babetie Bloch, Marin 
County CMHS. P.O. Box 2728, San Rafael, CA 94912, 
(415) 499-6785; or, Ralph W. Accardi. Member, Marin 
County Mental Health Advisory Board. 



services, in order to represent the needs of older 
adults in those specialized areas of planning. 

EVALUATORS OR MONITORS 

Planning task forces are established for six to 
eight weeks each year. Task force members are 
drawn from the community at large and from 
standing advisory committees. Professionals, lay 
people, consumers, and service providers gen- 
erally are represented, with the objective of 
having broad and balanced representation. There 
is some continuity of task force membership from 
year to year, and some new members are added. 
CMHS staff, MHAB members and staff or board 
members of agencies that contract with CMHS 
may not be voting members of a planning task 
force. However, all meetings are open to the 
public, and time is provided for the participation 
of nonmembers. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The State requires community involvement in 
the annual planning process. This county's CMHS 
also has a policy to provide maximum opportunity 
for public participation in the development of 
mental health services. Planning task forces help 
to implement this policy by making recom- 
mendations based on needs assessments, studies of 
service delivery, and evaluation of the progress 
made towards implementing the previous yet ^'s 
plans. 



LEVEL OF PARTICIPATION 

Task forces are charged with: (1) assessing the 
level of service being provided, (2) assessin^i the 
degree to which services meet existing needs, 
(3) identifying unmet needs, and (4) identifying 
special problem areas or barriers to service 
implementation. Task force members collect and 
analyze all necessary information needed to make 
the assessments and identify the sources from 
which this information will come. Task force 
efforts are assisted ably by CMHS staff. The final 
report is written by the task force. Standing 
committees review the final reports and make an 
independent evaluation of the recommendations. 
OASAC may support task force recommendations 
or may add recommendations of its own. 
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PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Planning task forces address the entir.e spec- 
trum of mental health services needed by older 
adults in Marin County. They also assess how .veil 
these needs are being met and what further 
acti'^ns are needed to improve the service de- 
livery system for this "^^^rticular group. The task 
force's focus may shift from year to year if a 
particular need appears to be pressing. All of the 
above issues require assessment of available or 
needed private sector services as well as service 
provided by CMHS. 

TECHNIQUES USED 

In order to formulate a reasonably accurate 
picture of the service system as it exists, a 
variety of data sources are used. These include 
county demographic data, statistical case data, 
presentations by informed professionals active in 
t>"-* field of mental health, experiences of "self- 
help" groups and other human service agencies 
and organizations. 

FINDINGS OF EVALUATION 
OR MONITORING 

Without exception, the evaluation portion of 
the planning process has shown a real commit- 
ment on the part of the mental health staff to 
provide quality services within the limits of its 
resources. The use of a citizen's planning team is 
viewed positively by the director and staff. 
Consequently, response to the task forces' reports 
and recommendations have been good. Some 
significant changes and improvements have been 
brought about in patterning the total mental 
health delivery system on recommendations made 
by planning programs, e.g., mobile geriatric 
evaluation team, establishment of geriatric and 
minority service coordinator positions, etc. 

RECOMMENDATIONS 

Recommendations listed in the report for the 
1982-83 planning year included: 

• Maintaining linkages throuyhout the com- 
munity with other providers of services to 
older adults 

• Monitoring the service utilization rates of 
older adults and attenipting to increase 
utilization of mental health services 



• Formulating an overall gerontology training 
plan for various audiences including staff 
from acute care hospitals, skilled nursing 
facilities, and CMHS 

S TEPS TO ENSURE IMPLEMENTATION 

The planning task force's report is carefully 
prepared and documented to support its recom- 
mendations. Information and ideas from staff are 
solicited along the way to ensure that proposed 
actions are reasonable and attainable. Copies of 
the draft service plan are made available to task 
force members. They evaluate the plan against 
the recommendations made and have the op- 
portunity for additional suggestions. The OASAC 
makes its position known with respect to the 
recommendations. The draft plan is then pre- 
sented to the mental health advisory board at a 
public hearing, at which the planning task force is 
represented. 

EXTENT OF IMPLEMENTATION 

It must be recognized that this is a continuing 
process of planning and evaluation from year to 
year, rather than a specific project with finite 
parameters. Most recommendations made by 
planning task forces have been accepted in prin- 
ciple. When resources permit, they have been 
incorporated immediately into the mental health 
service plan. When resources are insufficient, 
recommendations are not discarded by are listed 
as "unfunded services" for possible future im- 
plementation. These unit recommendations are 
carried forward from year to year in the planning 
process if the need still exists. 

RESOURCES AND COSTS 

Paid staff time is approximately 80 hours per 
task force per year. Paid clerical support is 40 
hours. Volunteers donate 160 hours and CMHS 
provides staff support services. 

ADDITIONAL COiVIMENTS 

A process that includes public participation in 
developing an action plan tends to carry more 
credibility and acceptance. Furthermore, as more 
and more people become involved in the planning 
process from year to year, the level of awareness 
within the community will increase, providing 
support for continued funding of mental health 
services. 
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40. CMHC Board Members Incorporate Evaluation Results 

injo Agency Planning 

David L. Silver and Donald A. Craig 

Northeast Kingdom Mental Health Service 



SUMMARY 

As part of its evaluation responsibilities, the 
program and evaluation committee of the North- 
east Kingdom Mental Health Service (NKMHS) 
board of directors conducts annual interviews 
with individual staff members who represent the 
agency's various services and support systems. 
The results of these interviews together with 
other program information are presented to the 
board of directors in the form of specific recom- 
mendations and are also included in the agency's 
annual evaluation report. Moreover, the com- 
mittee's recommendations become incorporated 
into the agency's policy and planning process. 

T YPE OF ORGANIZATION 

The program anu evaluation committee, onR of 
several committees of the NKMHS board of 
directors, is composed of up to eight members — 
two-thirds of whom are from the agency's gov- 
erning board. Committee members are appointed 
by the board president and serve from one to 
three or more years. 

The committee functions under periodically 
reviGVi^ed policies and procedures; its four primary 
responsibilities include the following: (1) to 
establish long range agency goals and priorities; 
(2) to review and evaluate existing programs on an 
ongoing basis; (3) to assist in the development uf 
new programs; and (4) to meet with staff mem- 
bers annually and make recommendations to the 
board accordingly. 

EVALUATORS OR MONITORS 

The program and evaluation committee pre- 
sently consists of the following: a former district 
school administrator, now administrator of an 
extended care facility; the administrator of a 
convalescent center; a former mental retardation 
professional, now owner of a general store; a 
former social worker, now a museu n curator; a 
retired businessman; and a practicing attorney. 

REASONS FOR THE EVALUATION 
OR MONITORING 

Evaluation has been an important part of 

For further tnfcrmation write Don?ld A. Craig, 
Planrer/Evaluator, Northeast Kingdom Mental Health 
Servio \ P.O. Box 724, Newport, Vermont 05855, (802) 
334-6744. 



Northeast Kingdom Mental Health Service 
(NKMHS) activities since the early 1970s. Board 
member involvement in evaluation has developed 
gradually and was formally incorporated into the 
committee structure of the board of directors 
with the enactment of the Community Mental 
Health Centers Amendments of 1975 (Public Law 
94-63 ). By 1977, the agency bylaws were 
amended to include a program and evaluation 
committee. Since that time, annual agency 
evaluation reports have been prepared with 
assistance from the program and evaluation 
committee. As defined by NKMHS staff and board 
members, the agency's evaluation process is not 
related so much to research as it is oriented to 
pragmatic identification of, and solutions to, 
problems, and is an essential part of the agency's 
planning cycle and management practices. 

LEVEL OF PARTICIPATION 

As part of the agency's evaluation process, 
committee memberb are involved in developing 
interview guides, conducting interviews with a 
variety of knowledgeable individuals (key in- 
formants), including program staff under contract 
to NKMHS, and other related activities. Much of 
the program and evaluation committee's work, 
however, is done with the assistance and cooper- 
ation of the agency's administrative staff. For 
example, committee members and staff conduct 
monthly meetings throughout the mental health 
service area concerning evaluation and planning 
issues. 

TARGET OF EVALUATION 
OR MONITORING 

The agency's ongoing evaluation efforts are 
focused on its internal performance and on the 
mental health needs of the catchment areas. 
NKMHS serves a poor, rural area thai encom- 
passes three counties with a total population of 
"^5,000. Thi. area has consistently ranked at or 
near the top in statewide needs assessments. Even 
though the area served by NKMHS has certain 
documented needs, the agency is providing the 12 
servii.3 elements and the compliance features 
that were mandated by P'jblic Law 94-63. The 
agency has inpatient service contracts with two 
smali general hospitals, but separate psychiatric 
units within those tacilities have been dis- 
couraged. The btate hospital, which may be a 
distance of 30 to 100 miles away, provides backup 
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services for the acutely ill and severely chronic 
patients. For general mental health services, 
NKMHS employs the only full-time psychiatrist in 
the area, who is a member of the National Health 
Service Corps. Some contracted part-time psy- 
chiatric service is also available to the agency. 
Although private practic*^ in mental health pro- 
fessions—other than psychiatry —is growing, it is 
evident that such services will not meet area 
needs in the foreseeable future. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Evaluation studies conducted by committee 
members have addressed a number of concerns: 
citizen awareness and satisfaction, consumer 
satisfaction, unmet needs, setting priorities for 
services, performance, public image, utilization 
of emergency services, utilization of psychotropic 
drugs, adequacy of facilities, and others. 

TECHNIQUES USED 

The primary techniques used to evaluate 
specific agency concerns were: (1) questionnaires 
are mailed to board members, staff, and other 
providers, including area physicians (responses are 
designed to assure anonymity and to encourage 
additional comments); and (2) interviews are 
conducted annually with selected staff. The 
questionnaire consists of a series of open ended 
questions, several av;c.reness and acceptability 
questions and a scale with which to rate the 
performance of the agency. 1 its interview guide, 
committee members ask staff associated with 
each agency program to provide information on 
such issues as the quantity and quality of services, 
the effectiveness of various treatment modalities, 
the strengths and weaknesses of existing pro- 
grams, and recommendations for program im- 
provement. Staff responses are supplemented with 
information on units of service, unit costs, and 
budget performance. 

FINDINGS OF EVALUATION 
OR MONITORING 

Since the evaluation findings may vary from 
year to year, this section presents a summary of 
selected questionnaire responses for FY 1982. For 
example, board members, staff, and providers 
were asked to rate the agency's performance and 
public image on a scale from 1 to 10. Based on a 
combined average rating of all respondents, the 
agency's overall performance was ranked higher 
(8.0) than its public image (6.9). In terms of these 
two indicators, the three respondent groups did 
not differ significantly in their responses; how- 
ever, NKMHS staff did rate their performance 
and public image higher than the other two 
respondent groups. 

In addition, board members, staff, and other 



providers were asked to rank 14 programs, serv- 
ices, or activities oy priority. Based on a com- 
bined ranking of all responses, the top three 
priority areas included: outpatient counseling, 
emergency services, and alcohol programs. 
Although specialized childrens' services ranked 
fourth among all of the combined responses, they 
were one of the services with the greatest dis- 
parity between respondent groups, e.g., board 
members ranked them socond, staff third, and 
providers tenth. 

RECOMMENDATIONS 

For FY 1982, the committee presented two sets 
of recommendations to the NKMKS board. Based 
on information collected through the question- 
mires, the following types of recommendations 
were made: (1) future survey instruments should 
be more precise and should include sampling of 
client and citizen opinions; (2) the program and 
evaluation committee should meet with the 
board's public eaucation committee to explore 
ways'of promoting prevention programs; and 

(3) efforts to educate the public regarding agency 
prevention programs should be intensified. 

The committee's recommendations, on the basis 
of staff interviews, included, but are not limited 
to, the following: (1) an employee assistance 
program, with a prepared timetable, should be 
actively pursued; (2) the lack of privacy in waiting 
rooms should be addressed; (3) steps should be 
taken to ensure that a backup is available at all 
times for receptionists who do client intake; 

(4) billing policies and procedures should be 
rf viewed; (5) vocational education should be 
included as an optional client service in the 
training-in-living-skills day treatment program; 
and (6) utilization of specialized foster parent 
homes should be increased. 

STEPS TO ENSURE IMPLEMENTATION 

The committee reviews its recommendations at 
every monthly meeting to ensure that they have 
been carried out and incorporated into the 
ongoing planning functions of the agency. In 
addition, the recommendations are discussed in 
one or more of the weekly NKMHS administrative 
staff meetings. The center's executive director 
will also respond to the committee either verbally 
or in writing regarding the implementation of its 
recommendations. 

EXTENT OF IMPLEMENTATION 

With respect to the recommendations made in 
FY 1982, the following events have occurred: (1) a 
timetable has been adopted for the employee 
assistance program; (2) steps have been taken to 
correct the lack of privacy in NKMHS waiting 
rooms and to provide backup staff for recep- 
tionists; (3) billing and vocational training issues 
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are still under study, but a number of staff hours 
have ah-eady beer .%pent on developing improve- 
ments, e.g., a new fee schedule will soon go into 
effect; and (5) increased utilization of specialized 
foster parent homes has been accomplished, thus 
increasing the viability of that program. 

SPECIAL BARRIERS OR SUPPORTS 

A lack of resources constitutes the greatest 
barrier to the implementation of evaluation 
activities. At the same time, certain Federal and 
State agencies continue to require evaluations 
and, therefore, have prompted agency board 
members and staff to identify and pursue evalu- 
ation activities. In addition, monitoring the 
implementation of evaluation results is difficult 
because limited resources constrain implemen- 
tation of some projects and more pressing 
problems divert the attention of agency staff. 
Committee members, however, have been willing 
to travel throughout the service area to ensure 
that their evaluation results are carried out. 

One particular problem in using staff interviews 
for evaluation purposes is the potential risk of 
entangling individual personnel matters in the 
process. In 198?, a special effort was made to 
avoid personnel issues. Moreover, there is an 
active NKMHS personnel committee, and a rou- 
tine grievance procedure available to any staff 
person. 



RESOURCES AND COSTS 

In 1982, each member of the program and 
evaluation committee contributed an average of 
about 85 hours or a total of 510 hours to evalua- 
tion activities. Clerical and administrative staff 
time spent on evaluation related issues amounted 
to another 160 hours; the NKMHS planner/ 
evaluator also spent some of his time providing 
necessary statistical information to committee 
members. 

ADDITIONAL COMMENTS 

The sense of importance and accomplishment 
that is felt by committee members helps 'iustain 
their high level of commitment to, and involve- 
ment in, the evaluation process. Moreover, their 
direct contact with staff during the process helps 
to strengthen staff -board relationships. 

Policy and planning formulation resulting from 
evaluations or from problem resolution is a com- 
plex process in an organization where partici- 
patory democracy is encouraged and where the 
organization must respond quickly, flexibly, and 
effectively to changes in its operating environ- 
ment. Ideally, evaluations or problem resolutions 
should result in the development of policies that 
can be used more than once, thereby reducing the 
energy required by the agency to handle recurring 
problems. 



41. A Key Informant Survey to Determine Accessibility and 
Acceptability of Mental Health Services 

Kevin M. Kindelan, George F. Mailly, and Kathy B. Hayes 

Winter Haven Hospital Commmity Mental Health Center 



SUMMARY 

In 1981, the program evaluation and research 
committee of the Winter Haven Hospital Com- 
munity Mental Health Center (WHHCMHC) 
conducted a needs assessment study using a key 
informant approach. Information was obtained 
from 49 individuals regarding their perception of 
mental health problems and needs within the 
community as well as their assessment of the 
visibility, acceptability, and accessibility of the 
center's services. A final report presented the 
rationale for the study, the methodology, the 
results, and specific recommendations to the 
WHHCMHC's director and the coordinator of the 
hospital's quality assurance department. 



For* further information write Kevin M. Kindelan, 
CMHC, Winter Haven Hospital, Winter Haven, FL 
33881, (813) 293-1121, ext. 1363. 



TYPE OF ORGANIZATION 

The needs assessment study was conuucted by 
the program evaluation and research committee 
of the Winter Haven Hospital CMHC. The 
WHHCMHC is a comprehensive community men- 
tal health center that receives funding primarily 
through State and local resources. It began 
operations in 1967 and was the first community 
mental health center in Florida. The CMHC is 
part of a general hospital and is served by the 
hospital's board of directors. 

EVALUATORS OR MONITORS 

The program evaluation and research com- 
mittee is composed of tnree professionals em- 
ployed full-time by the center: two psychologists 
and a social worker. The responsibilities of the 
committee are threefold: (1) to advocate for 
evaluative and research activity with the 
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WHHCMHC, (2) to provide necessary evaluative 
data for the WHHCMHC, and (3) to serve as 
consultants for staff who are performing exhaus- 
tive or research studies. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The needs assessment study, conducted in the 
fall of 1981, was part of an ongoing evaluative 
effort initiated by mental health center staff. For 
example, in 1980, a continuity-of-care study and 
a consumer-satisfaction study were also 
completed. Up until the current study, the most 
recent needs assessment data for the WHHCMHC 
was from an epidemiological survey completed in 
1974. 

LEVEL OF PARTICIPATION 

The program evaluation and research com- 
mittee was responsible for planning, designing and 
conducting the survey, as well as analyzing the 
data for the needs assessment study. In addition, 
the committee was responsible for writing and 
distributing the final report. In this effort, citi- 
zens are involved as solicited reactors in that 
their views and concerns regarding the existing 
local mental health system were incorporated into 
the study. 

TARGET OF EVALUATION OR MONITORING 

The needs assessment study focused primarily 
on the area surrounding Winter Haven, Florida, 
and the multiple services offered by the center. 
Winter Haven and the environs have a population 
of approximately 100,000. There is a sizable 
population of school age (approximately 24 per- 
cent of the Dopulation is under the age of 14) and 
about 45 percent of the population falls between 
the ages of 25-64 years. 

PROBLEMS O R ISSUES EVALUATED 
OR MGNITO; l 

This study had four primary objectives: (1) to 
determine the area's mental health related prob- 
lems and service needs, (2) to assess the visibility 
of existing center services, (3) to assess the 
acceptability of existing WHHCMI IC services, and 
(4) to assess the accessibility of existing 
WHHCMHC services. A secondary objective was 
to assess the respondents* satisfaction with the 
needs assessment questionnaire. 

TECHNIQUES USED 

The technique utilized was a key informant 
approach. The key informants were identified by 
the program evaluation and research committee 
as individuals in the community who were aware 
of its mental health problems and needs. A total 
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of 49 out of 56 key informants contacted actually 
participated in the study. These individuals, who 
were nearly equally divided between males and 
females, rr»presented a wide array of human 
service delivery and other specialties. For 
example, some key informants worked in the 
areas of elementary and junior high guidance 
counseling, social services, ministry, law en- 
forcement, employment counseling, hospital 
administration, homemaking, and county health 
department nursing administration. 

Each of the 49 key informants was mailed a 
packet containing a letter describing the study 
and requesting their involvement, and the needs 
assessment questionnaire. The questionnaire was a 
22-item instrument, derived from other sources 
and adapted for the center by the committee. An 
administrative staff member contacted each key 
informant soon after the packet was received in 
order to schedule a telephone interview. Most of 
the informants were interviewed over the phone 
by a member of the program evaluation and 
research committee, but a few mailed in their 
questionnaire without an interview. 

FINDINGS OF EVALUATION OR MONITORING 

Key informants identified two problem areas of 
greatest concern: substance abuse (i.e., alcohol or 
drug abuse); and family issues (i.e., family prob- 
lems, marital problems, child abuse and spouse 
abuse). The mental health service reported to be 
most needed was public information about mental 
health. Classes in job-seeking skills, crisis inter- 
vention, parenting skills, and services to abused 
victims also rated highly in terms of perceived 
need. 

Most of the key informants were aware of the 
center's services. A service with high visibility 
was the center's 24-hour telephone counseling 
services for the elderly. In terms of the accessi- 
bility of services, the key informants indicated 
that a major problem was the "stigma" associated 
with receiving mental health services. Other 
significant problems related to accessibility 
included transportation, wages lost or time off 
from work, and the cost of services. The attitudes 
of staff and the location and appearance of the 
facilities were infrequently mentioned as reasons 
for inaccessibility. In terms of acceptability of 
the center's services, there was general sat- 
isfaction with the staff's response to a referral. 
Some key informants, however, were dissatisfied 
with the lack of feedback on a referral from 
center staff. 

RECOMMENDATIONS 

As part of its final report, the prog»*am evalu- 
ation and research committee submitted seven 
recommendations based on the needs assessment 
study to the Winter Haven Hospital CMHC's 
director. The recommendations included four 
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areas that should be considered by the center: 
(1) providing divorce-support groups; (2) providing 
classes in step-parenting skills; (3) offering 
outpatient services during the evening hours; and 
(4) reviewing the policy and procedures regarding 
mental health professionals who continue to work 
with a former or current client who is admitted to 
the inpatient s'jite. The remaining three recom- 
mendations were oriented m';re toward action. 
They included: (1) developing programs to foster 
constructive community attitudes regarding 
emotional problems and obtaining mental health 
services; (2) developing a workshop that would 
focus on referral procedures and issues (i.e., 
referring the resistive or apprehensive clie.it, 
initiating feedback on clients referred for serv- 
ice); and (3) developing a forum for the mutual 
exchange of ideas between inpatient unit staff 
and area mental health professionals. 

STEPS TO ENSURE IMPLEMENTATION 

The final report and the seven recommen- 
dations were submitted by the Winter Haven 
Hospital CMHC's director to the task force on 
center policies, an interdisciplinary advisory 
group composed of trfc;atment and administrative 
staff, and to the coordinator of the hospital's 
quality assurance department. During one of the 
weekly meetings of the task force on center 
policies, the report and recommendations were 
presented and discussed by the chairman of the 
program evaluation and research committee. 

EXTENT OF IMPLEMENTATION 

Although the task force on center policies did 



not adopt any specific implementation plans for 
the recommendations, it is hoped that the final 
report and recommendations will provide a basis 
for future administrative decisions or program- 
matic changes. 

SPECIAL BARRIERS OR SUPPORTS 

No major barriers to, or supports Tor, the 
evaluation effort were present. 

RESOURCES AND COSTS 

The three members of the program evaluation 
and research committee spent approximately 
120 hours over a 4-month period in designing, 
implementing, and writing up the study. , pprox- 
imately 20 hours of clerical assistance were used 
to set up interview times and to type the various 
materials and final report. Key informants 
contriouted approximately 37 hours, or about 45 
minutes individually, to the study. 

ADDITIONAL COMMENTS 

A future refinement of the study will be to 
utilize a specific sample of key informants and to 
inquire about their perception of a limits ' range 
of mental health issues. For example, a sa.nple of 
citizens involved primarily with children, 
adolescent, and family issues (i.e., parents, 
teachers, guidance counselors, judges, law 
enforcement personnel, and others) could serve as 
key informants. These persons could be asked to 
identify the mental health needs and problems 
within a specified category (i.e., children, par- 
enting, etc.). 



42. Volunteers Assess; the Feasibility of Closing 
a State Psychiatric Center 

Wiiiiam P. Benjamin 

Central Islip Psychiatric Center Board of Visitors 



SUMMARY 

In 1982, the Central Islip (New York) Psychi- 
atric Center board of visitors conducted an 
evaluation of a proposal to close the State 
facility. Board members examined the need for 
the center and how the patients could be cared 
for in other facilities. The board presented a 
position paper to the New York office of mental 
health that included the board's recommendations 
and rationale regarding the closing of the State 
hospital. 

For further information write William P. Benjamin. 
Deputy Director, Central Islip Board of Visitors, P.O. 
Box 233, Smithtown, NY 11787, (516) 360-5337. 
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TYPE OF 
ORGANIZATION 

The Central Islip Psychiatric Lc er board of 
visitors is composed of seven volunteers who are 
appointed by the Governor of the State of New 
York. Three of the seven members must be 
relatives of current or former patients. All of the 
board members reside in the catchment area that 
is served by the center. It is the responsibility of 
the board to monitor the quality of care of 
patients at the center. The board also has a 
responsibility to evaluate and to offer advice on 
any proposed change or alternative use for the 
facility. 
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EVALUATORS OR MONITORS 

The assessment was conducted by the six 
members on the board of visitors (one position 
was vacant at the time of this study). Two of the 
members are social workers, one is a college 
professor, one is a Roman Catholic priest, another 
member is an executive with the telephone 
company, and one member is retired and the 
mother of a patient in the facility. 

REASONS FOR THE EVALUATION 
OR MONITORING 

In 1982, a proposal was made by the township of 
Islip and the New York Institute of Technology to 
purchase the 550 acre campus where the psychi- 
atric center is situated. The backers of the 
proposal wanted to convert the facility and 
property into a college campus and to locate 
high-technology industry ad jr cent to the college. 
This plan was well received by the community 
since it held out the possibility of revitalization 
for the lown. The plan, however, called for the 
displacement and movement of the patients at the 
facility. As such, the board members determined 
that an evaluation of *'he feasibility of such a 
proposal was warranted. 

LEVEL OF PARTICIPATION 

The board members were involved in varying 
degrees of participation in this evaluation, i.e., 
collecting data, conducting interviews, and 
preparing written reports. The board decided that 
since it was representative of the community, an 
ad hoc committee be established that would 
consist of the various factions that would be 
affected by the proposed change. Therefore, 
employees, patients, and other community 
residents became part of the ad hoc committee. It 
was the purpose of the committee to supply data 
concerning the issue of closing ths hospital to the 
board and to assist the board in analyzing the data. 

TARGET OF EVALUATION OR MONITORING 

The evaluation focused on the care provided to 
more than 2,000 patients at the psychiatric 
center. The evaluators looked at the type and 
level of care received by the patients as well as 
the components of care. 

The catchment area for the Central Islip 
Psychiatric Center is th'^ county of Suffolk with a 
population of more than 1,300,000 persons. In 
addition to the psychiatric treatment available at 
the center, the facility also contains a 99-bed 
medical and surgical hospitaL This hospital serves 
not only the 2,000 patients at Central Islip 
Psychiatric Center, but also the medical and 
surgical needs of three other regional , ^chiatric 
centers. These other centers have a combined 
population of 6,000 patients. The facility also has 
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a number of outpatient community clinics that 
are scattered about the country. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The evaluation study analyze the possibility of 
patients receiving the same level of care in other 
facilities or programs and the availability of beds 
for such care. Further, the evaluation studied the 
effects of relocation and transfer on patients, and 
the need for a medical surgical hospital in the 
region. 

TECHNIQUES USED 

Interviews were conducted with a variety of 
key infor'^ants, or knowledgeable individuals, for 
the feasioility study. Board members and others 
interviewed patients and families regarding their 
feelings about the hospital and their ability to 
accept relocation. A demography, study was also 
conducted regarding the accessibility of com- 
munity services for patients who would be 
relegated. Staff associatfid with various com- 
munity and advocacy groups were contacted in 
order to ascertain their interest in serving this 
patient group. In addition, other facilities in the 
region were surveyed to assess service availability. 

FINDINGS OF EVALUATION OR MONITORING 

The board members and others found that most 
of the patients, especially the elderly patients, 
feared relocation and transfer. Most of them 
looked at relocation in terms of losing their home. 
Moreover, the consensus of the interviewees was 
that moving the elderly patients would have a 
traumatic effect on their psychological well- 
being. It was also found that there was a dearth of 
nursing home beds and family care beds in the 
community to serve the geriatric population. In 
terms of the medical-surgical services issue, it 
was found that patients from other psychiatric 
centers were receiving inadequate c^re in com- 
munity hospitals for these prcblenrs; therefore, 
there was a continued need for a medical-suroical 
hospital to serve the r. gion. In the final analysis, 
the evaluation results suggested that while closing 
the facility might bring some economic advan- 
tages to the community, it would present some 
severe problems in terms of providing adequate 
patient care. 

RECOMMENDATIONS 

Based on the evaluation results, the board felt 
that the facility should rot close. The board, 
however, oid feel that the community's economic 
needs should also be addressed. In examining the 
physical plant and other structures on the hospital 
campus, the board concluded that the hospital 
could consolidate and utilize half of the campus 
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while selling the other half to the town and the 
college. In this manner, both the needs of the 
patients and the community residents would be 
served. 

STEPS TO ENSURE IMPLEMENTATION 

The board attempted to sell its plan to as many 
factions of the community as possible. For 
example, the board of other State hospital 
facilities as well as community and advocacy 
groups were contacted regarding the advantages 
and disadvantages of the board's plans. In addi- 
tion, the Governor and certain State legislators 
were contacted. 

EXTENT OF ir/IPLEMENTATION 

Although the board's plan was well roceived by 
other facility boards and by the State Association 
of Boards of Visitors, the State office of mental 
health did not accept the plan. It was evident that 
the office of mental health could not approve the 
board's plan since »t did not fully meet the needs 
of the State college. A compromise, however, was 
finally reached whereby the psychiatric center 
would continue to serve the geriatric population 
in the region and to provide medical-surgical 
services for those patients. The population of the 
facility would be reduced to 550 patientct l./j50 
patients would be transferred to other facilities in 
the region. Under this plan, the center would be 
consolidated, but not closed, and the town and 



college would purchase approximately 500 acres 
of the existing hospital property. 

SPECIAL BARRIERS OR SUPPORTS 

A great deal of support for the study came 
from the hospital staff and advocacy groups. A 
major barrier was the lack of understandir , /"the 
needs of the mentally ill, especially by groups 
such as the civic associations which were more 
interested in increasing real estate values. 

RESOURCES AND COSTS 

The boar J conducted this evaluation without a 
budget and, when necessary, absorbed those costs 
that did occur. In addition. State hospital and 
other agency staff donated their time to the 
board. Although the -otal number of hours spent 
on the evaluation is difficult to estimate, some 
board members donated over 300 hourr to the 
study. 

ADDITIONAL COMMENTS 

Because of its commitment to quality patient 
care and to the patients them.dlves, the board 
became a v*^ell-respected group within the com- 
munity. As a result of the evaluation, the board 
gained a great deal of visibility, and its ability to 
secure additional resources for the patients at the 
facility was enhanced. 



43. Emergency Services in High Point 

Barbara Geddie 

High Point Mental Health Asso -iaticn 



SUMMARY 

The High Point Mental Health Association 
(MHA) first identified the need for local 24-hour 
emergency services in its community in 1972. The 
High Point catchment area is unique in that it 
Includes the second and sixth largest cities in the 
State. As a resuit, t-he area requires a center that 
provides a full range of serv'^^es in each city, and 
an overall area office to direct and coordinate the 
two facilities. High Point citizens had to travel 20 
miles to the other center for emergency services 



For further information write Barbara Geddie, State 
Delegate, Mental Health *\ssoc. in North Carolina, 5 
West Hargett Street, S\iito 705, Raleigh, NC 27601, 
(919) 828-8145; or Louise '"•alloway, MSW, Director. 
Higji Point Division of Guilfort County Mental Health 
Center, 236 Boulev?-rd. High Point, NC 27262, (919) 
883-1341. 



when the local center was closed, and area office 
staff were reluctant to establish after-hours 
emergency services in High Point. The need for 
such services also was documented in 1975 by the 
United Way and the High Point MHA in a survey 
of community mental health needs, and by the 
local mental health center director who con- 
ducted a survey in 1981. Also during the years, 
the need continued to bs identified through the 
county advisory board and through MHA repre- 
sentatives at site reviews conducted by the 
ujtional Institute of Mental Health and the North 
Carolina Division of Mental Health. The county 
advisory board and MHA contacted county 
commissioners. A model for emergency services 
was presented to tie county commissioners, and 
via radio to the public with support by the local 
MHA chapter. The model was approved by the 
county commissioners and area authority and is 
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now being implemented. 

TYPE OF ORGANIZATION 

The High Point Mental Health Association is 
composed of catchment area residents who are 
interested in the delivery and quality of com- 
munity-based and institutional mental health 
services. The board of directors is elected from 
the membership, and two-thirds of the board must 
consist of persons not professionally engaged in 
the mentai nealth field. Employees of public 
mental health agencies are not eligible for board 
• embership. The three ex officio n^embers are: 
tne director of the High Point Merital Health 
Center; the county program director who is 
responsible for the catchment area program 
(including book centers); and a United Way liaison 
person. The board has representation according to 
race, sex, religious affiliation, and economic level 
consistent wiLi the characteristics of the catch- 
ment area. The board's role is to advocate for the 
mental health needs of local citizens, and to 
monitor and facilitate service quality and the 
center's responsiveness to community needs. The 
association makes recommendations directly to 
the High Point branch of the Guiiford County 
Mental Health Center, and indirectly to the 
county via representation on the advisory board to 
the county mental centers, to the area board 
(which is the mental health authority) and to the 
regional office of the North Carolina Division of 
Mentol Health through citizen participation in 
their onsite review team. 

EVAUJATQRS OR MONIT >^RS 

The evaluation wa:; c< d by the local 

MHA and the team was u of eight 

housewives, six clergy, t • dte practitioners 
(a psychiatric nurse and a psyohiatrist), one 
congresswoman, four educators, one doctor, one 
retired educator, two businessmen, and one 
graiuate student. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The continued monitoring was necessary 
because the area office remained reluctant for 10 
years to look seriously at expansion of emergency 
services locally. The local MHA chapter will 
continue to monitor the service as it expands. The 
inadequacy of emergency services after hours and 
on weekends surfaced when new commitment laws 
were passed and citizens requiring initial 
assessments had to be transported by the sheriff's 
department 20 miles to the emergency s^ ''vice 
unit at the second center. The result was that 
patients were detained for hours awaiting evalu- 
ation and proper disposition. The MHA maintained 
that though the service was available, it did not 
adequately meet standards for accessibility. 
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LEVEL OF PARTICIPATION 

The High Point Mental Health Association was 
involved in each phase of the initiation, surveys, 
and implementation of a model for expansion of 
emergency services. MHA members initiated and 
participated in a United Way survey to document 
need, wrote letters to county commissioners in 
support of the model, and continually raised the 
issue to keep it visible to mental health officials. 
The pressure kept up until there was sufficient 
support from authorities to implement the plan 
that the local director proposed and the associ- 
ation supported. 

TARGET OF EVALUATION OR MONITORING 

The agencies studied were the Guildford County 
Mental Health Center and loc^l division of the 
mental health center; and the specific program 
studied was emergency services. According to the 
1980 census, Greensboro has a population of 
155,642 and High Point 64,107 for a combined city 
population of 219,749. County residents outside 
each city are included in the catchment area and 
the total population in Guilford County is 317,154. 
Afhar-hours emergency services were primarily 
provided by the county emergency unit. There was 
no local psychiatrist in private practice; a 
psychiatric nurse consulted with local physicians 
to facilitate private emergency evaluations. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITOR ED 

This monitoring encountered a number of 
related problems — inaccessibility, delays ^n 
evaluation and treatment, transportation prob- 
lems to emergency services and to the hospital, 
reluctance of magistrates to issue a commitment 
petition, and reluctance of emergency room 
physicians to either initiate a petition or com- 
plete the first medical evaluation recommending 
commitment. 

TECHNIQUES USED 

Interviews were conducted with: magistrates; 
selected patients who had usc:d emergency 
services and who were in the socialization 
program operated by the association; the director 
of the outpatient and emergency departments of 
the local hospital; local police; school counselors; 
clergy; and local physicians. The purpose of the 
interviews was to determine local and area 
responsiveness and document problems in delivery 
of emergency services after hours. Local and 
county emergency service staff were interviewed 
during the State onsite visit. 

FINDINGS OF EVALUATION OR MONITORING 

Board members were pleased with reports from 
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the above-mentioned referral sources, which 
substantiated that local ennergency services were 
responsive c-nd effective during hours, and that 
the difficulties lay in getting local patients to the 
after-hours ennergency service 20 nniles away. 

RECOMMENDATIONS 

Based on the two surveys of needs and on 
interviews, the MHA board reconnnnended that an 
alternative plan to expand ennergency coverage in 
High Point be developed, and that statistical 
information about after-hours contact and 
systematic followup be instituted. 

STEPS TO ENSURE IMPLEMENTATION 

The local center director was already eager to 
expand emergency coverage. Once the issue was 
Identified and recorded in the onsite review, and 
letters of support were received by the county 
commissioners, the local director had the backing 
needed to present a model for expansion to the 
area boE:*d and commissioners in order to secure 
funding for a modest beginning toward expansion. 
Progress was monitored through MHA repre- 
sentation on the area board and through frequent 
contact with the center director. 

EXTENT OF IMPLEMENTATION 

The center is currently negotiating wit^ two 
nurses who will be employed for after-hours 
coverage with backup from the local center 
psychiatrist. This nurse will provide liaison 
between the mental health center and the hospi- 
tal, and will conduct evaluations in the emergency 
room. After consulting with the oncall psychia- 
trist, the nurse will direct the patient to the 
proper agency whicri will then coordinate and 



expedite further treatment in an appropriate and 
timely fashion. 

SPEC I AL BARRIERS OR SUPPORTS 

The Jocal center director and emergency 
services staff were very supportive of MHA's 
project and helped direct its members through the 
steps necessary to attain approval. Barriers 
included budget cuts that severely limited funds 
for expansion and the desire of area prcg'^^m staff 
to contain after-hours services in the adjacent 
city. 

RESOURCES AND COSTS 

United Way absorbed the costs and provided 
volunteers for their study. Individuals from the 
association contributed their time and expenses 
related to their documentation efforts. The local 
center provided clerical and technical assistance 
for their study and the resulting model for 
expansion. The association did not have to 
allocate money from its budget. 

ADDITIONAL COMMENTS 

Although this project re'^resents a lengthy 
process, it was facilitated by the board's per- 
sistence and the local director's expertise and 
commitment. The alliance between the center and 
the association and the dedication to this proj^^ct 
were very effective in working through resistance 
and generating a creative model for expansion of 
a vital service durir , a time of severe budgetary 
cuts. Furthermore, an essential element was the 
careful identification of the political elements in 
the environment of a v^enter that had to be 
attended to. 
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How Will You Go About It? 



Having defUifid their problem and specified a series of research 
questions that warrant further investigation, Doug Brown now has to 
identify a strategy Ice answering these questions. At least three factors 
have to be considered by the committee before they can select a method- 
ology. The first is the nature of the committee's authority with the mental 
health system. Groups that have formal authority o^er facilities within the 
system clearly have more latitude regarding the types of demands for data 
that they can make. Those without a formal mandate have to rely on the 
voluntary cooperation of agencies or, failing this, collect data that are not 
dependent on agency cooperation (e.g., information that is part of the 
public record or information ""rom key informants who are not affiliated 
with the agency or agencies bv, ig evaluated). 

The second factor that will constrain Doug Brown's choice of a 
method is, of course, the nature of the problem being evaluated. Those 
problems that are very narrow and clearly definec frequently can be ad- 
dressed with a single approach. For example, client satrfaction with 
services can generally be assessed adequately with a client survey, and the 
issue of staff morale can be evaluated with a survey of personnel attitudes. 
Doug Brown's problem, however, is quite broad and touches on a variety of 
different issues. His group, therefore, will have to design a strategy that 
includes a number of different approaches or methods. 

The third consideration that will constrain the group's choice of a 
methodology is the level of resources at the conimittee's disposal. Such 
resources include funding, availability of data, expertise of committee 
members, availability of outside expertise, and the amount of time that 
committee members can afford to donate to the project. If these f<xctors 
are not accurately assessed, the group runs the risk of beginning an overly 
ambitious project that they may be unable to complete. 

Having considered the various research methods available to them, 
Doug Brown's committee selects a methodological approach that will auow 
it to explore several aspects of the problem and that will make full use of 
available resources. Since the decision to close a wing of the Riverview 
State Hospital has been a controversial one within the community, the 
committee decides to begin with a limited number of public forums in 
which citizens, as well as mental nealth professionals, will have an oppor- 
tunity to make their opinions known. 

Secondly, the committee plans to use data that have already been 
collected by the State and r junty departments of mental health. Typically, 
mental health departments have some information on the characteristics of 
clients residing in Stale institutions aiid on the number and types of pro- 
viders within the community. These data will allow the committee to 
assess tfc match between the needs of the clients who are to be discharged 
to the cOi^^munity and the services that are available in the community. In 
the present case, the committee is fortunate in ha\'ing a relationship with a 
local community college and will be able to rely on some student and pro- 
fessional assistance in analyzing these data. 

Recognizing the wealth of professional expertise and knowledge 
about this commimity's needs and resources among local service providers, 
the committee decides to augment their data collection efforts by 
interviewing knowledgeable providers. Finally, one nr.omber of the 
evaluation committee who i?: a certified public accountant will examine 
available data on the costs c providing services to clients i-i the institution 
and in the community. 

The committee recognizes that this is a very full research agenda and 
has allocated a full year to completing the various components. 



The cases in this section illiistrate a variety of 
different research methods that can be iised by 
citizens in evaluating or monitoring mental health 
services. In many instances, those methods can be 
carried out without a great deal of technical 
assistance, and none of the methods illustrated 
here requires a substantial budget. 

Site Visits aiid Key Informant Interviews 

The approach that is descr^jed most frequently 
in other chapters of the Casebook is a combin- 
ation of site visits and interviews. Some evalua- 
tors conducted site visits only. Typically in these 
caises the evaluators visited programs using some 
form of checklist. They observed the agency's 
Of^eratiot^ and physical facilities and recorded 
S(;ores within predetermined categories. In other 
Cc\ses the evaluators combined the observational 
data with data obtained from interviews with 
staff and/or clients. 

Secondary Data Analysis 

The first case included in this chapter, reported 
by Frances P. Meehan and John J. McDonough, is 
an example of a secondary data analysis tech- 
nique. In this case, members of the Los A.ngeles 
County Mental Health Advisory Board were at- 
tempting to put pressure on the county adminis- 
tration to apponit a permanent director of mental 
health. In order to buttress their arguments, 
members of the advisory board obtained and re- 
viewed data collected by the county. They used 
salary data and information contained in auditors' 
reports to the State. In addition, advisory board 
members also conducted interviews with a number 
of individuals in key administrative positions. The 
combination of secondary data analjrsis and key 
informant interviews is an ei-ective approach 
that is used frequently in evaluation research. 

Case S^ulation 

The second case is described by Aldene Hart 
and Barbara Goza. In this case, members of a 
citizens' advisory council evaluated a commuiaty 
mental health center's crisis services through a 
case simulation technique. Three advisory council 
members placed a series of afterhours telephone 
calls to the center's crisis unit posing as clients in 
need of aissistance. Crisis service staff were una- 
ware of the research project until the study was 
completed. This case represents a study that was 
well designed and carefully executed. The data 
collected were appropriate given the questions 
that the citizens wanted to address. However, 
such clandestine methodi should be used cau- 
tiously. Had the results shown the service to be 
poor, tne secretive nature of the research meth- 
ods could have had negative repercussions on the 
board's relationship with center staff. This is not 
to say '^at such measures should not be used-- 
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only that the potential value of the information 
obtained should be carefully weighed against 
possible negati/e fallout. 

Clinician Interviews 

In the third case, reported by Elizabeth Fulton 
and Bruce Hirsch, members of a mental health 
association were interested in exploring the 
match between the needs of clients being referred 
from the area's psychiatric emergency services 
and the services that were available in the ccrzi- 
munity. To address this question, members con- 
ducted a series of interviews with referring 
clinicians in the psychiatric emergency service 
units. Prior to the actual data collection, the 
volunteers met with the directors of each psy- 
chiatric emergency service and visited the fa- 
cility during daytime and evening hours. As a 
result, they were aware of how the units 
operated and were able to design an interview 
schedule that focused on relevant issues. This 
type of preparatory effort is crucial in designing 
useful survey forms. It should be noted that this 
interview method can also be used to collect data 
from other types of mental health providers, 
clie.:ts, clients' families, etc. 

Structured Group Approaches 

The next two cases in this section utilize 
structured group approaches. In one. Myrtle C. 
Nash and Sharon F. Bock describe the use of the 
"nominal group approach" in a needs assessment 
sttady. For this study, members of the program 
evaluation committee assembled four groups of 
citizens representing consumers, civic leaders, 
mental health professionals, and business and in- 
dustry. Members of each group were asked to 
identify unmet service needs in the community. 
The nominal group approach is a method that at- 
tempts to obtain maximum particip;»tion from all 
grc'^p members by focusing their attention on 
clearly defined issues and soliciting the ideas f 
each member in a structured fashion. Kate ri. 
Lothrop and Barbara Whetstone describe the use 
of a similar method, the Delbecq Nominal Group 
Technique, in their account of a study examining 
the issue of no-shows in a community mental 
health center. These methods are useful because 
they allow researchers to obtain feedback from a 
variety of sources in a very efficient manner. 

The case described by Hays and Whetstone also 
illustrates the use of a citizen review group. The 
1975 Amendments to the Community Mental 
Health Centeni Act mandated that citizens be 
involved in reviewing CMHC evaluations. Joan 
Zinober and Nancy Dinkel (1981) attempted to 
formalize this process throu^ a research project 
funded by the i<^tional Institute of Mental Health. 
The project ri:u'*tlted in a manual that describes 
tht process 0^" / evelop'ng citizen review groups 
for \ie purposes of rev.ewing agency evaluations 
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and making recommendations based on these 
secondary data analyses. 

Use of Standardized Instruments 

Rick Kastner and Marilyn Lee Olds describe a 
more standardized technique in their account of 
the Lancaster County Mental Health/Mental Re- 
tardation Board's use of PASS — Program Analysis 
of Service Systems. PASS is a method developed 
by Wolf Wolfensberger and Linda Glenn (1975) for 
evaluating residential programs for develop- 
mentally disabled persons. The method involves 
the use of structured interviews and checklist 
forms that cover program administration, fiscal 
matters, programmatic issues, and the physical 
and social environments of programs. This pro- 
gram is quite comprehensive and, since it involves 
the use of standardized instruments, researchers 
can compare their findings with published data on 
similar programs. This is certainly a useful at- 
tribute. Potential users of PASS, however, should 
be aware that the instniment is based on the 
ideological principle of normalization. It confers 
higher scores on tiiose programs that come closest 
to approximating a normal community enroll- 
ment for clients. For those programs tiiat do not 
share this commitment, another instrument might 
be more re>vant. 

Evaluability 

The next case in this section is an example of 
an evaluability ??tudy Vera Mellen describes a 
board of directors' effort to conduct a client 
outcome study. The citizens involved in this 
project began what they thought was to be a 
fairly straicjitforward outcome study and found 
that the d*^^ they needed were unavailable. In the 
end, they performed what was essentially an 
evaluability study, lajring the groundwork for 
further research by specifjring the data that would 
be necessary to conduct an adequate exploration 
of client outcomes. Evaluability studies are small 
studies that are conducted prior to large-scale 
evaluations. The objective is to assess the ade- 
quacy of available data and the overall feasibility 
of launching the larger evaluation. An evaluability 
study can be quite cost effective in those in- 
stances where the usefulness of existing info/- 
matica is doubtful. 

Freedom of Information 

The final case in this section is quite different 
from the preceding cases. In this report, Richard 



Hessler and Michael Walters describe a citizen 
group's attempts to obtain copies of NIMH reports 
concerning a local mental heJth. Cv^ter. The 
group felt that the data in thes^; reports were 
crucial for their ovm study of fcUowup s^^rvices 
for adult patients dibcharged to the commuinity. 
The group requested the duta from NIMH, the 
NIMH regional office, the State division of nxental 
health, and the center itself. Their initial at- 
tempts met with failure since these reports were 
exempt from the mandatory disclosure provisions 
of the Public Information Act. Eventually the 
group contacted the assistant chief of the Legis- 
lative Services Branch of NIMH requesting a 
formal amendment to the Public Information 
regulations that would give citizen groi^)s access 
to NIMH reports of site visits to menta^ health 
centers. The case was reviewed by the DHHS 
Freedom of Information Officer. Ultimately, the 
citizen group obtained the reports they were 
seeking and also succeeded in having the regu- 
lation changed so that other citizen groups would 
be guaranteed access to chis informrtion. The 
case clearly illustrates the power that well- 
organized citizen groups can have. 

Additional Methods 

A number of cases that appear in other chap- 
ters of the Casebook illustrate additional methods 
that citizens might use. One innovative sti'ategy 
is the use of computer modeling for resource 
allocation (case #31). Another method is record 
review in which the evaluators examine client 
records to extract the relevant information on 
clients (cases #20, #34, and #49). Client data can 
also be collected through interviews with clients 
as illustrated in cases #8, #9, #12, and #35. 
Finally, cases #7 and #18 illustrate the u;e of 
public forums for gaining data and ideas from a 
wide cross-section of the community. 

These cases illustrate the variety ot methods 
available to citizens. The methods discussed, 
however, are hardly exhaustive. Readers who de- 
sire more detailed descriptions of the m*»thods 
described here or other methods that can be used 
should refer to: Citizen Evaluation of Mental 
Health Services, by Val. D. MacMurray et al. 
(1976); Evaluation in Practice: A Sourcebook of 
Program Evaluation Studies for Mental Health 
Care Systems in the United States, edited by 
Gerald Landsberg et al, (1979); and A Working 
Manual of Simple Program Evaluation Techniques 
for Commvv.ty Mental Health Centers, by Homer 
J. Hagedom et al. (1976). 
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44. A Mental Health Advisory Board's Use of Secondary 

Data Analysis Techniques 

Frances P. Meehan and John J. McDonough 

Los Angeles County Mental Health Advisory Board 



SUtS/MARY 

In 1977, mental health services in Los Angeles 
County were provided by the department of 
health services. After extensive effort by the 
mental health advisory board, the Los Angeles 
County board of supervisors was persuaded tc 
separate the department of mental health from 
the department of health services. The board of 
supervisors appointed a new director in order to 
establish adequate levels of accountability for the 
county's mental health programs. 

TYPE OF ORGANIZATION 

Under California law, the mental health 
adv.3ory board is appointed by the Los Angeles 
board of supervisors to advise the board of 
supervisors and the director of mental health 
services. All members serve in the public interest 
and may not be employees of the county mental 
health department or of the agenc: ^s receiving 
Short-Doyle funds. The board must represent 
psychiatry, psychology, psychiatric nursing, 
psychiatric technicians, social work, the county 
supervisors, and the general public. 

EVALUATORS OR MONITORS 

The mental health advisory board was composed 
of a psychiatrist, another physician, a psychiatric 
tech«iic:an, a nurse, a social worker, a psycholo- 
gist, a County Supervisor, and eight additional 
members representing the general public. 

REASONS FOR THE EVALUATION 
OR MONITORING 

In 1974 tht department of mental health had 
been merged with three other county departments 
to form a department of health services. 

A general hiring freeze within the county was 
extended to personnel paid out of Short-Doyle 
funds, despite the fact that the State had already 
allocated these funds to the county. Requests 
went unheeded tc the interim acting deputy 
director of the department of mental health, to 
the director of health services, and to the 
director of the department of personnel for hiring 
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replacements with available Short-Doyle money. 
The result was that mental health programs were 
unable to spend the dollars allocated by the State 
and the county was able to divert the "savings'* 
into other programs. 

The need was acute for an extremely compe- 
ls )it. permanent director for mental health 
services. 

LEVEL OF PARTICIPATION 

The Los Angeles County mental health advisory 
board, a citizen group, became actively involved 
in the attempt to improve accountability and 
responsiveness within the mental health advisory 
system. These actions were independent of any 
government entity. 

TARGET OF EVALUATION OR MONITORING 

The focus of the board's efforts was the Los 
Angeles mental health program. The Short-Boyle 
mental health services program was located 
within the Los Angeles County department of 
health services. The director of health services 
was appointed by the board of supervisors as the 
local director of mental health services. Medical 
responsibility was delegated to the deputy 
medical director of mental health services. 

Los Angeles County has a population of more 
than seven million persons spread across a great 
diversity of communities and geographical areas. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The problems addressed by the advisory board 
were: 

• The need to establish clear channels of 
accountability for the effective delivery of 
county mental health programs 

• The need for information regarding the gross 
''savings*' associated with the county mental 
health freeze in 197 -77 and the net 
"savings," an explanation for the deteriora- 
tion of services despite a substantial increase 
in funding and a detailed breakdown of 
cen*''*al office costs 

• The need for a commitment that mandated 
Short-Doyle programs would not be seriously 
impede^ by the county personnel freeze 



136 

143 



• The need to give top priority to recruiting 
and selecting a permanent deputy director 
for mental health services and to upgrading 
the position 

TECHNIQUES USED 



explanation of the discrepancies between 
these two figures 

• An assessment of the roliability of da^ 
indicating a serious deterioration in the 
provision of services despite increased 
funding and a plan for i'Tiproving the 
information system 

• A break down of the $6.6 million central 
office costs including separate accounting for 
each major program element 



Members of the advisory board regularly 
attended the weekly policy meeting of the top 
mental health administrative staff and reviewed 
auditors* reports to the State. A member of the 
board requested salary data from regional 
directors. Interviews were held with the director 
of health services, the interim deputy director of 
mental health, program managers, regional 
directors, and the director of the department of 
personnel. 

FINDINGS OF EVALUATION OR MONITORING 

After reviewing available data and conducting 
numerous key informant interviews, members of 
the advisory board concluded that there was a 
need to separate the department of mental health 
from the department of health services. They also 
concluded that Los Angeles County had tried to 
save money by retaining a temporary interim 
deputy director of mental health who was unable 
to cope with extremely serious problems and that 
Short-Doyle money was being diverted from 
mental health to other needs of the department of 
health services. The accountabiUi/ of the board 
of supervisors was challenged. 

RECOh/W^ENDATIONS 

In 1977, the total funds expended on Short- 
Doyle programs for Los Angeles County was $130 
million with 90 percent provided by the State and 
a 10 percent match by the county. The Los 
Angeles mental health advisory board by law had 
the power to approve the planning process and to 
mak recommendations as to specific items in the 
plan. In June 1977, the Los Angeles County 
mental health advisory board unanimously decided 
that they would approve the planning process only 
if the. department of health services met a set of 
conditions specified by the board. The conditions 
included: 

1. Convening a •*summit'* meeting with county 
department of health personnel, repre- 
sentatives from the board of supervisors and 
the mental health advisory board in order to 
clarify accountability with respect to the 
delivery of mental health services 

2. Providing tha mental health advisory board 
with information on: 

• The gross savings associated with the 
1976-77 personnel hiring freeze, the net 
savings resulting from the freeze, and an 



3. Changes in organizational structure and 
priorities including: 

• Assigning top priority to recruiting and 
selecting a permanent deputy director for 
mental health services 

• Upgrading the role of the deputy director 
within the department 

• Upgrading the role of the regional mental 
health chiefs to include regional planning and 
policy formulation and clarifying the 
relationship between the regional chiefs and 
the department mental health chief 

STEPS TO ENSURE IMPLEMENTATION 

By a unanimous vote, the mental health 
advisory board refused to approve the 1977-78 
Short-Doyle planning process for mental health 
services beyond a conditional 90-day period (from 
the date of the approval of the board of super- 
visors). State funding is not forthcoming for 
mental health services without the advisory 
board's approval. 

EXTENT OF IMPLEMENTATION 

Following the mental health advisory board's 
report, a number of activities was initiated. The 
department of mental health was separated from 
the department of health services. A search for a 
permanent director of the department of mental 
health was initiated. Such changes have led to a 
more effective administration of mental health 
and increased respect for the mental health 
advisory board. 

SPECIAL BARRIERS OR SUPPORTS 

The administration of the health department 
was unresponsive to requests for information. 
Tracking expenditures through the maze of the 
health department budget was difficult. Regional 
deputy directors of mental health programs 
cooperated with the efforts by the mental healtii 
advisory board. They were willing to answer 
specific questions about expenditures to a citizen 
member of the advisory board. The mental health 
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advisory board members worked well together and 
persisted until ail changes were made. 

RESOURCES AND COSTS 

Some mental health advisory board members 
donated several hours per week over a period of 
several months. Each member opent at least 
several hours per month. All time and out-of- 
pocket expenses were donated. The executive 
assistant assigned by the department provided 
technical assistance. 

ADDITIONAL COMMENTS 

Our advisory board simply got tired of being a 
••rubber stamp** for decisions already made by 
health department bureaucrats who had no real 
interest in, or knowledge of, mental health 
problems within the county. We took a stand and 
made a vow to either get results or to resign. We 



fcuna the perfect vehicle. We were able tc use 
our power of approval over the planning process 
to virtdally stop the flow of State money and to 
get some action from the county. 

The power we had went beyond fiscal power. 
We discovered we had the ability to use planning 
authority to take important issues of mental 
health programs, effectiveness, and account- 
ability to the public and the media. 

In hindsight, we were fortunate to get every- 
thing we asked for and, at the same time, to 
enhance our standing with the county board of 
supervisors. The board of supervisors had taken an 
active role and has been int jsted in the mental 
health program ever since. 

You cannot underestimate the power citizens 
have in addressing program deficiencies and 
organizational problems. Citizen volunteers' 
independence and credibility have the potential 
for a great impact on the issues. 



45. Citizens' Advisory Council Uses Case Simulation 
Technique to Evaluate Crisis Services 

Aldene Hart 

Granite Community Mental Health Center Citizens* Advisory Council 

Barbara K. Goza 

Salt Lake County Division of Mental Health 



SUMMARY 

In response to a National Council of Community 
Mental Health Centers (NCCMHC) survey 
suggesting deficiencies in CMHC crisis services, 
the citizens' advisory council conducted a study 
of Granite Community Mental Health Center 
(GCMHC) crisis services. Three council members 
made 25 after-hours calls to the center's crisis 
service during a 3-week period. Fictitious names 
and vignettf 3 were used, and the staff were not 
informed of the study until data collection was 
completed. The advisory council expected lo find 
the GCMHC crisis service working pretty weli, 
and were not greatly surprised by the study's 
positive findings. The answering service response 
wi.s quick and courteous and ti crisis workers' 
responses were equally quick. 

TYPE OF ORGANIZATION 

The evaluation was conducted by the citizen's 
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advisory council for the community mental health 
center, in collaboration with center staff and 
management. The 20-person council was com- 
posed of interested corrnnunity members ap- 
pointed by the council with the approval of the 
county commission that serves as the governing 
board of the center. The advisory council 
considers itself an advocacy group—both for the 
center and for the community it serves. Mem- 
bership on the council is determined by profes- 
sional skills, special interests such as service 
consumer or minority representation, or general 
interest in mental health issues. 

EVALUATORS OR MONITORS 

The three advisory council members who 
performed this evaluation were lay persons. 
However, they did receive some consultation from 
the research and evaluation staff of GCMHC. 



REASONS FOR THE EVALUATION 
OR MONITORING 

The findings of a National Council of Com- 
munity Mental Health Centers survey suggested 
that CMHC crisis services in general were not as 
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effective as the community needed. The NIMH 
study had telephoned crisis numbers and found 
that one-third did not answer. In addition, there 
had been criticism of local crisis services. The 
director of GCMHC provided the advisory council 
with this information and the advisory council 
initiated the reported research in their role as 
program advocates. Many of the advisory council 
members had been on the crisis service:) team and 
were concerned about the way these services 
were being evaluated externally. They also felt 
that negative findings about GCMHC were 
unlikely in view of their experience. 

LEVEL OF PARTICIPATION 

The advisory council was totally responsible for 
the conduct of this research. They initiated the 
research, informing only the center director, 
adult and family unit director, and crisis service 
coordinator. The research staff provided some 
consultation in the design of the research and in 
analyzing the data. 

TARGET OF EVALUATION 
OR MONITORING 

Granite Community Mental Health Center's 
crisis service uses trained volunteers (all with 
professional backup) or mental health personnel in 
its after-hours crisis service. The crisis service is 
under the administrative supervision of the adult 
and family outpatient services at GCMHC, and is 
supervised by a crisis service coordinator. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The question to be answered by this research 
was: how accessible is GCMHC crisis service to 
individuals requesting services after normal 
working hours? 

TECHNIQUES USED 

In a 3-week period 25 after-hour calls were 
made to GCMHC crisis service. Except for 
several times specially chosen as the hardest 
times to get service (weekends or shortly before 
or after working hours), all call days and times 
were randomly chosen. The study team designed 
vignettes based on their crisis experienced at 
GCMHC and another local agency. When the 
answering service operator answered the phone, 
the researchers gave the operator a fictitious 
name, presented a problem, and asked to speak to 
a crisis we*i<er. Typical crisis vignettes were: 

• My daughter is talking about suicide. 

• The police are looking for my daughter and I 
would like ner taken to the mental hospital 
rather than to the detention center. 



• I have to have some holp because I'm afraid 
I'm going to hurt my little girl. 

• My husband just walked out on me. 

After the researcher spoke with the crisis 
worker, the worker was thanked, the study was 
explained, and the worker was asked to keep the 
study confidential. Data collected included the 
number of rings required to reach the answering 
service and the number of minutes to reach the 
crisis worker. 

FINDINGS OF EVALUATION OR MONITOP ^*^'*" 

The answering service response was quick and 
courteous. The phone rang more than 6 times only 
once (41 rings), and the average number of rings 
was 4.1 (median, 2.3). In only one case did the 
researcher consider that the answering service 
response might have been more caring: a 7:45 
a.m. call was responded to by asking the caller to 
wait a few minutes and call the center. It was 
also discovered that the 4 1 -ring call was probably 
due to a structural problem. The center had only 
two lines at the answering service, and when they 
were both busy, the line kept on ringing. 

The crisis workers' responses were equally 
quick. On 87 percent of the calls, the researcher 
was held on the line and then connected directly 
with the crisis worker. If the time on hold was 
more than a couple of minutes, the answering 
service operator checked back to reassure the 
"crisis case." The average amount of time until 
the researcher reached a crisis worker was 4.4 
minutes (medan, 1 minute). In 56.5 percent of the 
calls, the wait was 1 minute or less. The wait was 
longer than 10 minutes in only two cases. One call 
did not result in a crisis worker contact. Later 
investigation found that the crisis worker was 
given the wrong number by the answering service. 
Advisory council members also reported their 
qualitative impressions of tite crisis service. 

RECOMMENDATIONS 

The findings and recommendations of the 
advisory council study were reviewed by the 
center executive committee and were provided to 
NIMH at their yearly site visit, and to the 
national council offices. The major recommen- 
dation was that the Granite CMHC staff and 
answering service should be rewarded for their 
high-quality crisis service. The qualitative and 
quantitative data led the researchers to conclude: 
"We feel very strongly that this is exceptional 
work and shows the quality of Granite's crisis 
service. We also would like to commend the crisis 
staff and the center for their openness to 
evaluation, their lacK of defensiveness, and their 
understanding about being awakened at 5 o'clock 
in the morning." 

It was suggested that the answering service be 
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counseled about the importance of double- 
checking phone numbers to ensure their accuracy, 
and that a busy signal be arranged to indicate that 
both lines were busy. These suggestions were 
implemented. 

STEPS TO ENSURE IMPLEMENTATION 

The research team was particularly concerned 
with maintaining the objectivity of the study. In 
order to maintain this objectivity, crisis service 
staff were not informed of the research. The only 
staff members who knew about the research were 
the center director, the adult and family services 
director, the crisis service coordinator, and a 
research and evaluation design consultant. As 
crisis workers were informed of the study and its 
purposes, they were asked to maintain the 
confidentiality of the research. This confidence 
was maintained. 

EXTENT OF IMPLEMENTATION 

The advisory council felt that all recommen- 
dations had been implemented. 

SPECIAL BARRIERS OR SUPPORTS 

The research team considered the most 



important element supporting utilization of the 
research to be the upfront preparation to 
maintain the quality of the research itself. 
Several preparatory meetings were held to ensure 
the plausibility of crisis vignettes, to create a 
broad range of crisis problems in the vignettes, 
and to ensure random selection times for the 
crisis calls. Management was also extremely 
supportive of the rescsarch, and was willing to 
maintain the secrecy of the design. 

RESOURCES AND COSTS 

The amount of time required to perform the 
evaluation totaled 199 hours. The breakdown of 
hours were: advisory council members, 100 hours; 
research staff, 75 hours; crisis and adult and 
family directors, 4 hours; crisis workers (as 
subjects), 20 hours. 

ADDITIONAL COMMENTS 

The advisory council was very impressed with 
the crisis system, and with management's 
responsiveness to suggested changes. They did not 
feel that they would change any aspect of the 
research. Their primary lesson from performing 
this evaluation is: preparation pays. 



46. Mental Health Association Evaluation Using 
Interviews with Clinicians 

Elizabeth Fulton and Bruce Hirsch 

Mental Health Association of San Francisco 



SUMMARY 

This study of crisis services was conducted by 
volunteers from the Mental Health Association of 
San Francisco (MHASF), a volunteer nonprofit 
organization. The project grew out of a prelim- 
inary study of the psychiatric emergency services 
(PES) at two major San Francisco hospitals. The 
purpose of the crisis study was to assess the 
availability and appropriateness of followup 
treatment services for people who experience 
psychiatric crises and enter the system through 
PES. It sought to find out if there is a match 
between the clinicians* recommendations for care 
and available services. Where such a match did 
not exist, it attempted to identify the problems. 

TYPE OF ORGANIZATION 

The Mental Health Association of San Francisco 
is a nongovernmental, nonprofit volunteer or- 



For further information write Sharon George, 317 
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ganization affiliated with the Mental Health 
Association of California and the National Mental 
Health Association. It advocates for improved 
mental health services and seeks to educate the 
public about mental health and mental illness. Its 
800 members reflect the cultural diversity of the 
San Francisco community. The association is 
governed by a board of directors elected from the 
membership. 

EVALUATORS OR MONITORS 

This study was carried out by the public affairs 
committee of MHASF in cooperation with the 
directors of the psychiatric emergency service 
units being studied. The committee organized a 
group of 25 volunteers to conduct the stud>. The 
volunteers included students, housewives, and 
representatives from a variety of occupations. 
MHASF staff provided support to the volunteers. 

REASONS FOR THE EVA LUATION 
OR MONITORING 

The project grew out of a preliminary study of 
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the psychiatric emergency services at two major 
San Francisco hospitals. For many people, PES is 
the entry point to community mental health 
services programs. The preliminary study famil- 
iarized volunteers with PES facilities through site 
visits and identified the need for a study of the 
foUowup treatment services available to people 
who enter the mental health system through PES. 
The aim of the crisis study was to determine 
whether needed foUowup services were available 
in the community. 

LEVEL OF PARTICIPATION 

Before the data collection began, the public 
affairs committee drew up a formal research 
proposal that was approved by the department of 
public health. The 23 mental health association 
volunteers conducted 130 hours of interviews with 
clinicians. Each of the volunteers attended a 
3-hour orientation session and a 3-hour debriefing 
session. Mental health association staff prcided 
support to volunteers as part of the ongoing public 
affairs program. Volunteers worked with PES 
directors to draw up a questionnaire that meas- 
ured the appropriateness of referrals as assessed 
by the referring clinician. 

TARGET OF EVALUATION OR MONITORING 

The study involved two PES units that are part 
of community mental health services in the city 
and county of San Francisco. The facilities are 
the San Francisco General Hospital PES and the 
Westside Crisis Clinic at Mt. Zion Hospital. San 
Francisco General Hospital is located in an ethnic 
low-income area. It covers two catchment areas 
and handles overflow from other districts. The 
predominant groups using services of this crisis 
unit are white, black, Hispanic and Chinese. The 
facility is open 24 hours a day and handles 500 to 
800 patients a month for formal evaluations. The 
patients present problems across a wide range of 
crisis and dysfunctional states. The majority of 
patients are from the chronic psychiatric popu- 
lation and are diagnosed as psychotic or bor- 
derline psychotic. Most patients stay at thi^ PES 
from 1% to 5 hours. Inpatient referrals are to two 
inpatient psychiatric wards, the forensic unit of 
the medical/psychiatric ja'l ward at San Francisco 
General Hospital itself, private psychiatric wards 
at five other hospitals, or to the State mental 
hospital in Napa. The hospital has a transitional 
extended emergency service program for repeat 
users who do not follow through on community 
referrals. 

Westside Crisis Clinic is located in a pre- 
dominantly low-income area, although its catch- 
ment area includes some high-income sections. 
Major client groups are black, white, Chinese and 
Philippine. Westside serves one catchment area 
through the community mental health services 
program. It operates 24 hours a day and handles 



ERIC 



about 500 patients a month. The bulk of patients 
seen are chronically mentally ill. Many patients 
are in a life crisis or are experiencing a combina- 
tion of acute family, drinking and drug problems. 
The length of stay is from 1 hour to 3 daySr 
Inpatient referrals are made to private psychi- 
atric hospitals; State hospitalization is avoided 
where possible. Westside Crisis Clinic has a 
medication clinic for outpatients who do not 
follow through on referrals to community 
resources. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

PES is the entry point to the San Francisco 
public mental health treatment system for many 
people experiencing a psychiatric crisis. The study 
sought to find out if there is a match between 
clinicians* recommendations for care and avail- 
able services. Where such a match did not exist, 
the study attempted to identify factors that 
interfere with the referral process. Some of the 
factors examined were the type of diagnosis, 
whether the facility to which the referral was 
being made was at maximum capacity, and the 
degree to which regulations such as geographic 
districting limit the availability of services. The 
study examined how often each patient had been 
seen in the crisis unit, and compared the referring 
clinicians* preferences in terms of foUowup 
services with what services, if any, were available 
in the community. 

TECHNIQUES USED 

Volunteers met first with the administrators of 
each PES to become familiar with the operation 
of the units. They than visited the facilities as 
teams during key hours of the day and night. This 
experience gave volunteers some idea of the 
problems regularly encountered in the crisis units. 

With the aid of the PES directors, volunteers 
designed interview forms for clinicians that 
included demographic data for each clinician 
including: ethnic group, language capabilities, 
age, sex, professional training, amount of pro- 
fessional experience, and length of time at PES. 
Questions regarding diagnosis, the appropriateness 
and availability of foUowup services, and other 
factors Influencing referrals were also included. 

The method used in this study was to interview 
clinicians in the crisis units after the 'clinicians 
had evaluated cUents. There was no direct 
contact between volunteers and clients, and 
demographic information on clients was coded for 
hospital staff access only. 

Clinical diagnosis and a DSM III (Diagnostic 
Statistical Manual) code were entered on the 
interview forms. This permitted subsequent 
assessment of clinician attitudes toward avail- 
ability of services for clients with similar 
diagnoses. 
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During one week, 25 volunteers spent 1,300 
hours interviewing clinicians to connpile the data 
for the final report. Originally the data were to 
be analyzed manually, but the complexity of the 
study necessitated computer assistance. 

FINDINGS OF EVALUATION 
OR MONITORING 

The study uncovered a complex set of issues 
ranging from initial diagnosis to referral. Some of 
these had to do with the availability of services 
for referrals, and the difference between the 
actual referral made and the clinician's detei- 
mination of the ideal referral. In addition, some 
political, economic, and social constraints on the 
use of the appropriate referrals become apparent. 
The complexity of these issues required the 
assistance o^ specialists who are now conducting 
further computer-assisted data analyses in order 
to compile mere specific findings. 

RECP' ^^NDAIIQNS 

The Hnal repc t on the study is not available 
yet. F<ecommendations that have been made to 
date concern additional analyses tc be conducted. 
The analysis will be a collaborative effort be- 
tween the mental health association and 
specialists in research methods and statistical 
techniques. Specific issues being analyzed are: 
(1) diagnosis and clinician's confidence in the 
diagnosis; (2) contributing problems and secondary 
diagnosis; (3) ideal disposition for the patient; 

(4) whether the required services exist and 
whether the patient was sent to that services; 

(5) why some patients were not sent to services 
that were available; (6) the clinician's evaluation 
of the actual referral as opposed to the ideal one; 
(7) how the referral was made; and (8) how many 
times the patient had been seen in the unit in the 
last 6 months. 

The University of California at San Francisco 
found the preliminary data interesting enough to 
provide a small biomedical support grant to the 
director of one PES. Further analysis of the data 
is now undenway at the university. 



STEPS TO ENSURE IMPLFMFNTATION 



The project is not yet completed and, there- 
fore, the implementation plan has not been 
finalized. 

SPECIAL BARRIERS OR SUPPORTS 

Initially, the clinical staff at the crisis units 
had reservations about the presence of volunteers 
in the units, since some volunteers were un- 
familiar with the realities of PES work. However, 
the volunteers were sensitive to the needs of staff 
and to the necessity for evaluating what took 
place without interfering. In the end, the process 
was one of mutual education for volunteers and 
staff alike. The orientation for volunteers, led by 
PES officials, was very important in building 
bridges between the two groups. PES directors 
were particularly instrumental in making the 
volunteers feel welcome. 

RESOURCES AND COSTS 

Hospital visits were conducted by 25 volunteers 
over a period of 7 days, totaling 130 hours of 
observation and interviews. Volunteers were 
supported by mental health association staff as 
part of the MHA's ongoing public affairs program. 

ADDITIONAL COMMENTS 

The study gave volunteers an opportunity to 
understand the central place occupied by PES 
units in community mental health services. Since 
PES units are often the first contact patients 
have with the system, the appropriateness of 
foUowup services is an important subject of 
investigation for professionals and volunteers 
alike. 

The volunteers were impresfod with the 
perscnal and professional dedication of clinicians 
in the PES. They also felt that since their study 
obtained information never available before, it 
would increase the ability of the crisis units to 
serve people entering the treatment system 
through these facilities. 
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47. Citizens Participate in a Nominal Group Approach 

to Identify Needs 

Myrtle C. Nash and Sharon F. Bock 

Spartanburg Area Mental Health Center 



SUMMARY 

The Spartanburg Area Mental Health Center's 
(SAMHC) program evaluation committee (PEC) 
conducted a key informant needs assessment 
survey utilizing the "Nominal Group Approach." 
Citizens who were knowledgeable about the 
community were asked to identify the area's most 
pressing unmet needs. Four separate workshops 
were convened for: (1) agency representatives, 

(2) spokesmen for the minority community, 

(3) representatives from industry, and (4) clients. 

TYPE OF ORGANIZATION 

Evaluation activities for the center are the 
responsibility of the program evaluation com- 
mittee (PEC), established as part of an ongoing 
quality assurance program during the late 1970s. 
It is a multidisciplinary committee, composed of 
center staff members appointed by the program 
director with the advice and consent of the 
medical director. The committee is responsible 
for all evaluation activities required by Federal 
and State funding sources including needs 
assessments, consumer satisfaction surveys, 
biometry and epidemiology reports, and various 
service component evaluations. It also functions 
as a resource to individual program managers who 
wish to undertake service element evaluations not 
necessarily required by external sources. Finally, 
the committee is responsible for overseeing the 
management of patient and fiscal data. 

The committee is accountable to the program 
director and the medical director and routinely 
reports evaluation activities to them. Required 
Federal and State reports are a matter of public 
record ana are included in the SAMHC's annual 
report. 

EVALUATORS OR MONITORS 

This study was conducted by SAMHC staff using 
citizens as key informants. The citizens who 
participated in the evaluation were chosen to 
represent different points of view. There were 
professionals (agency representatives), clients, 
civic leaders (spokesmen for the minority com- 
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munity), and representative from business and 
industry. The separate groups were selected with 
the expectation that each would tend to perceive 
the functions required of the center somewhat 
differei tly. We anticipated a wide range of views 
about h )w our agency is perceived in the com- 
munity and which needs it is failing to meet. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The State office requires an annual needs 
assessment. The evaluation team recognized the 
value of assessing a broad spectrum of the 
community, but resource limitations precluded 
the design of a c omprehensive evaluation tool and 
administration to a large sample; hence, the 
Nominal Group Approach was chosen as an 
alternative model that would provide information 
in an efficient manner- The Nomina* liroup 
Approach provided the center evaiuators with 
data anr suggestions from a cross section of 
citizens and was undertaken with a minimum 
expenditure of staff time and effort 

LEVEL OF PARTICIPATION 

The PEC was solely responsible for the devel- 
opment and implementation of the needs assess- 
ment project. Committee members selected the 
Nominal Group Approach, developed questions to 
be answered by participants and collected lists of 
citizens who would be representative of various 
target groups. With assistance from other staff 
members, the PEC conducted the needs assess- 
ment meetings. Each of the citizen participants 
attended one 3-hour workshop. The committee 
then compiled and analyzed the information and 
developed summaries for presentation to the 
center executive committee, the total r^^^^. and 
the board of d»^ectors, and for inclusion in the 
SAMHC annur evaluation report for 1979-80, The 
l(5tter of thar».o that went to each participant also 
included a summary of results. 

TARGET OF EVALUATION 
OR MONITORING 

The study was limited to Spartanburg County, 
one of the counties served by the Spartanburg 
Area Mental Health Center in Spartanburg, South 
Car'^lina. SAMHC is a comprehensive mental 
health center and provides the twelve services 
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that were rnandated by law at the time of this 
study. It has a psychiatric director, a program 
director, and an administrator or business man- 
ager. Each service has a coordinator. These 
coordinators, the three executives mentioned, and 
the administrative assistant to the business 
manager constitute the executive committee. 

The city of Spartanburg is a highly indus- 
trialized community in a rural county. Spar- 
tanburg started as a cotton mill town that now 
has widely diversified, mostly light industry. In 
the past, Spartanburg has sent a dispropor- 
tionately large number of people to the State 
hospital. There has been speculation that con- 
ditions in industrial plants might be a relevant 
factor. The estimated population of the city of 
Spartanburg is 43,000; that of the county is 
197,000. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

This evaluation focused on the quality of 
service. Three questions were formulated for use 
in the workshops: (1) "What are the most serious 
unmet needs for mental health services in the 
community?"; (2) '^What is the most serious 
barrier to getting help at the mental health 
center?**; (3) "What stories have you heard about 
the center which might make it hard for pome 
people to come here?- It turned out that only the 
first question could be thoroughly analyzed by the 
Nominal Group Method in the 3-hour workshop. 

TECHNIQUES USED 

The Nominal Group Method was suggested by 
the chairman of the evaluation committee, 
selected by the committee, and approved by the 
executive committee of the center. It was 
decided to use four sessions, each one for a 
different group of informants. 

Potential participants were identified by center 
staff. Letters of invitation were drafted and 
modified slightly for each of the four groups. The 
invitation described the goal of the project, 
outlined particulars of the workshop, and en- 
couraged participation. 

The actual conduct of the workshops is of some 
interest. After all participants hsd been wel- 
comed individualiy and then collectively by the 
center director or program director, the coor- 
dinator of the workshop briefly outlined the 
workshop coals and reviewed the format to be 
used. Participant? were then asked to break into 
small groups. The individual's group was desig- 
nated by a color on each name tag. The question 
to be addressed was on the blackboard. 

Each participant was asked to write down as 
many responses to the question as he/she could 
think of during a 10-minute period. Then the staff 
member began with one person and asked the 
person for one item from his/her list. The re- 
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corder wrote the item on a tablet on the wall. The 
next person was asked to contribute an item from 
his/her list. This sequence went on until every- 
body had a chance to contribute an item. Each 
group member was then asked to indicate which 
of the items listed on the tablet were also on 
their individual lists. A check mark was put on the 
item each time it was mentioned. Listing 
continued until all items from the lists were 
recorded and tallied. Ten minutes for discussion 
and clarification followed. Then a new sheet of 
paper was started to indicate combinations of 
items and relationships. 

After all possible combinations were listed, the 
participants were given a few minutes to select 
the top five prioricies and write them down 
individually. Then a vote was taken and the 
number of votes for each item entered on the list. 
Next a priorities Jist was made containing the five 
items receiving the most votes. 

After a short break, the workshop members 
workt^u together and the final list of priorities for 
the question was established. 

FINDINGS OF EVALUATION 
OR MONITORING 

Essentially, we found that the greatest 
perceived need for mental services was for public 
information about mental health and about the 
center. There was considerable agreement about 
this from all four groups of informants. A 
summary of reported needs in order of priority 
follows: 

Representatives from Agencies 
1. Primary prevention/education 
2.5 Service provider coordination 
2.5 Outreach to people whero they are 
(homebound, including terminally ill) 

4. Public relations 

5. Crisis intervention services 

Representatives from the Minority Community 

1. Pl V:c information and education 

2. Need for persons to identify with, e.g., 
black male counselors for black men 

3. Transportation 

4. Mental health outreach to community 
centers 

5. FoUowup on patients after treatment 

Repro&jntatives from Industry 

1. Public infOi-mation and and education 
("crash course** for recognizing mental 
healtn problems) 

2. Reaching specif ic industrial 
organizations, e.g., personnel/nurses 

3. Mental health field's knowledge of 
industry 

4. Schools 

5. Programs for abused spouses/children and 
abusers 
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Patients 

1. Stigma with mental health services; 
better public relations to reduce stigma 

2. Change name of center 

3. More programs dealing with stress 

A. Better community education programs 
5. More funding instead of less for mental 
health services 

RECOMMENDATIONS 

After reporting the findings of the needs 
assessment to the executive committee, the 
evaluation committee made its recommendations. 

In view of the progress in the area of con- 
sultation and education (C&E) that had been made 
under the leadership of the pai o-time coordinator 
of CAE, the evaluation committee suggested that 
the coordinating position be made fuU-time. When 
the issue was finally resolved, this person 
continued as a halftime coordinator and another 
worker was assigned halftime to consultation and 
education to work primarily with incjstry. 

STEPS TO ENSURE IMPLEMENTATION 

With the presentation of recommendations to 
the executive committee of the Spartanburg Area 
Mental Health Center, the formal authority of the 
evaluation committee ended. The information 
gathered in the four workshops was, however, 
further disseminated. All of our informants 
received a report on the results of the needs 
assessment. The results were reported at a 
meeting of the clinical staff of the center and 
summaries were posted on bulletin boards at the 
county courthousf., city hall, the county health 
department, the county library, and the menlai 
health center. The information was also dis- 
seminated by two members of the evaluation 
committee who were also members of the 
executive committee. 

EXTENT OF IMPLEMENTATION 

Because the data pointed so clearly in a single 
direction, we were able to make a single recom- 
mendation that was implemented in modified 
form. We asked for a full-time coordinator of 
consultation and education and got two halftime 



people. During the year of this evaluati'-n, the 
number of persons reached by consultation and 
education rose? from approximately 10,000 to well 
over 50,000. 

SPECIAL BARRIERS OR SUPPORTS 

The major constraint placed upon the PEC was 
a limitation in personnel resources. All five 
members " the committee had full-time 
responsibility for clinical and administrative 
activity at the center. As a result, the work of 
the comnirttee was an additional activity that 
placer! ar c.(tra burden on each n^amber. 

The principal support for the project was the 
autonomy given to the committee by the center 
director. Staff members were generous with their 
time in assisting with the workshop and secre- 
tarial support was available for the varied clerical 
tasks that were required. 

Although several strategies were used to get 
potential participants to come to the workshops, 
not everybody who was invited came. We do not 
know how much information we missed as a result 
of this. On the other hand, the Nominal Group 
Method did insure that all who participated had 
their opinions and priorities recognized. As a 
result, morale at all of the workshops was 
exceptionally high. 

RESOURCES AND COSTS 

It is estimated that the total expenditure of 
time, from conception through report preparation, 
amounted to ai proximately 30 workdays. Based on 
the mean salary of all individuals involved, the 
personnel costs were approximately $1500. 
Supplies and refreshments cost approximately 
$100. Not included in any of these estimates are 
overhead costs for the facility. 

ADDITIONAL COMMENTS 

In general, we are satisfied with the way this 
needs assessment was conducted. Since the 
Nominal Group Method enables one to get ideas 
and facts from all informants both in the initial 
expression of thtir opinions and in the ordering of 
priorities, it is uniquely valuable to the sponsoring 
organizations and to the participants. 
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48. Citizen Review Group Assessment of "No Show'' Rates 

at a New Hampshire CMHC 

Kata R Hey9 and Barbara W. Whatatone 

Central New Hampshire Community Mental Health Services, Inc. 



SUMMARY 

A citizen group that included former mental 
health center clients was formed by the board of 
directors of the Central New Hampshire Com^ 
munity Mental Health Services, Inc. (CNHCMHS). 
The group reviewed an evaluation that staff had 
made regarding clients who do not appear for 
their initial appointment. The group formulated a 
list of recommendations to decrease the ^no 
show** rate. The recommendations wero presented 
to the board of directors, and the board in turn 
requested comnr»*^nts and compliance by the 
agency staff. 

TYPE OF ORGANIZATION 

A citizen review group (CRG) composed of 
catchment area residents was formed to conduct 
a review of one of Central New Hampshire 
Community Mental Health Services* annual 
program evaluations and co make recommen- 
dations to the board of directors and staff. The 
board of directors authorized and selected a 
program evaluation to be reviewed, ami appointed 
two board members to form the citizen review 
group. 

EVALUATORS OR MONITORS 

The citizen review group was composed of eight 
citizens from the catchment area (2 male, 6 
female) who had been previous clients of 
CNHCMHS. Assistance was pnwided by two board 
members and two staff members (the center's 
program evaluation consultant and the progr^^m 
director for ^ -o program being reviewed). 
Occ jpations represented on the CRG were: 
secretary, city planner, engineer, teacher, 
student, and parents. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The Federal government mandated that mental 
health centers conduct seme form of citizen 
review of the center's evaluation as a require- 

For further information write Kate F. Hays, Ph.D., 
Twin Rivers Counseling Center, Franklin Regional 
Hospital, Aiken Ave., Franklin, NH 03235. (603) 
934-3400; or Barbara W. Whetstone, R.N.. M.S., Board 
of Directors, CNHCMHS, Inc., 740 N. Main Street, 
Laconia, OH 03246, (603) 528-3066. 



ment for Federal funding. The particular program 
evaluation reviewed was selected by the board of 
directors because it was considered to be a study 
worthy of review, having implications relating to 
community acceptance and revenue generation. 

LEVEL OF PARTICIPATION 

The eight citizens conducted the review and 
formulated the list of recommendations. The 
board members and one staff member met with 
the citizen review group. They pxuv'ided guidance 
and any information needed to accomplish the 
task. They also assisted in compiling recom- 
mendations. The other staff member was 
responsible for selecting and contacting members 
of the CRG, and describing the project to the 
them. 

TARGET OF EVALUATION OR MONITORING 

The program evaluation reviewed wac a study 
of the -no show** rate at the adult outpatient 
service (AOPS) at CNHCMHS. The adult out- 
patient service is located in Concord and serves 
aduU3 from the Concord catchment area. 
CNHCMHS as a whole served approximately 4,300 
people during the year of the review. Of all the 
clients seen 65 percent had an income of $ 10,00b 
a year or less. Fewer than 25 percent of the 
clients had a history of prior hospitalization. 
CNHCMHS provides inpatient, outpatient, partial 
hospitalization, emergency, drug and alcohol, 
consultation and education and supportive 
services. These programs serve people of all ages, 
in one urban central office and three rural branch 
offices. 

PROBLEMS OR IS f^ UES EVALUATED 
OR MONITORED 

A review was conducted of the AOPS study of 
clients who did not show up for their first 
appointments. The citizen review group focused 
on the inta;:e procedure used with new clients. 

TECHNJQUES USED 

The primary review method was the Delbecq 
Nominal Group Method. CRG members were given 
copies of the study to be reviewed and sufficient 
information to place the evaluation in the context 
of the center's entire functioning. A list of 
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recommendallons made by CRG members was 
discussed nnd then voted on. The recommen- 
dations raceiving the highest number of weighted 
votes were expanded to narrative form. A letter 
of recommendations was then sent to th3 board of 
directors. 

FINDINGS OF EVALUATION OR MONI'lORING 

In response to the center's concer • about the 
rate of "no shows" at intake, CRG members 
I'ocused their attention on the *nteraction 
between the potentio* client ^jncJ the initial 
telephone contact with the center prior to intake. 

RECO MMENDATIONS 

T^.d CRG made a number of recommendations. 
These included: 

• Potential clients should speak initially with a 
trained Intake worker. 

• The intake worker should take the initiative 
in offering Information and reassurance. 

• Background Information (such as a brochure) 
should be available to send to clients. 

• A foUowup call to the client should be made. 

• The best "match" between client and 
therapist should be arranged. 

• Training and a "refresher" course should be 
developed for intake workers. 

The CRG sent these recommendations to the 
board of directors. 

STEPS TO ENSURE IMPLEMENTATION 

The CRG recommendations were made to the 
board with the explicit expectation that there 
would be a written response from the board within 
6 months, and that the CRG would reconvene for 
a final session to review the board's response to 
the recommendations. This, in fact. Is what 
happened. 

EXTFNT OF IMPLEMENTATIO N 

The board sent the CRG's recommendations to 
all programs within the agency. IL found that most 
programs were already complying with the 
recommendations. In particular, \ system utilizing 
trained Intake workers had been operational since 



the time of Ihe study, and Intake workers were 
taking the initiative with regard to reassuranr<5 
and sharing Information. The recommendation for 
an agency-wlde brochure gave additional Impetus 
to an activity that had already been initiated. As 
followup calls had been found to have little 
impact on the "no show** rate, this recom- 
mendation was not Implemented. Staff develoD- 
ment plans for the next year Included specific 
training and refresher courses for intake workers 
throughout the agency. 

SPECIAL BARRIERS OR SUPPORTS 

The major difficulty In recruiting CRG 
members was getting enough names of former 
clients, and then getting a sufficient number of 
people to agree to participate. In addition, there 
was some attrition of group members during the 
course of the project. This was In marked contrast 
to other times when CRG's composed of "key 
informants" have been recruited. In those 
instances, recruiting had not been a problem. 

Primary supports for the project were an 
atmosphere of openness, curiosity, and accep- 
tance on the part of board and staff; CRG 
member enthusiasm for the process and ap- 
preciation of the request to be involved; and an 
especially effective .method (the Delbecq 
technique) of generating ideas from a number of 
people. 

RESOURCES AND COSTS 

Resources committed to the study were 
approximately: 

• Staff —evaluator and program director, 15 
hours clerical, 5 hours 

• Board— 2 board members, iO hours each 

• CRG~8 members, 6 hours each 

• Other Costs— writing board, file cards, pens, 
paper, refreshments 

ADDITIONAL COMMENTS 

The citizen review of the program evaluation 
process had a number of beneficial effects. The 
process of program evaluation was more credible 
because of the board's active involvement In the 
CRG thr »Mghout the project. In addition, the 
attention paid by the CRG to •'no shows" and the 
initial Intake process Increased staff interest In 
decreasing the "no show" rate. 
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49. Citizen Evaluation Using Program Analysis of 
Service Systems [PASS] 

Rick Kastner and Marilyn Lee Olds 

Lancaster County Mental Health/Mental Retardation Drugs/ Alcohol 



SUMMARY 

In 1972, the program evaluation committee 
(PEC), a volunteer citizen group that is a 
committee of the Lancaster Co'jnty (Pennsyl- 
vania) MentalHealth/Mental Retardation Board, 
was designated to work directly with the division 
of operations research to establish a procedure 
for review, comment, approval, and policy 
re;:cmmendations concerning all evaluations 
completed within the MH/MR system. In fulfiainq 
this responsibility, the committee adopted 
Program Analysis of Service System (PASS) 
(W'-.ifensberger and Glenn 1975) as one instrument, 
for program evaluation to be used in the com- 
munity for a 3-year period. By this action, the 
program evaluation committee (PEC) equated 
program quality wi'-h integration of the nor- 
malization principle. This is consistent with the 
goals and philosophy of the Lancaster County 
MH/MR program. PASS is a standardized 
evaluation tool of fifty ratings that evaluates a 
program, its administration, quality of the 
environment, and the degree to which the 
program integrates the clients within the 
community. In 1981, the program evaluation 
committee adopted the short form of PASS, an 
18-item evaluation tool, instead of the longer and 
more cumbersome 50-item form of PASS. This 
evaluation instrument is used to assess programs 
such as outpatient services, inpatient services, 
residential facilities, crisis intervention, case 
management services, day treatment facilities, 
and vocational rehabilitation. 

TYPE OF ORGANIZATION 

The MH/MR board is composed of citizens and 
includes representatives from the medical field, 
social work, nursing, education, religion, business, 
and one county commissioner. The county 
commissioners make the final appointments to the 
board upon the recommendation of the board's 
nomination and personnel committee. The 
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program evaluation committee, which oversees 
the evaluation process, is also composed of 
citizens who are recruited throughout the county. 
The duties of the MH/MR board include reviewing 
the performance of programs within the system, 
and development of policies and procedures to 
govern the programs. 

EVALUATORS OR MONITORS 

Each PASS team consist? of three to six persons 
including a team leader. All are trained citizen 
volunteers who have gone through 12 hours of 
instruction in order to learn how to use the 
evaluation tool. Th -^se volunteers include MH/MR 
consumers, business and industry representatives, 
persons with financial backgrounds, students, 
providers of services, representatives of advocacy 
organizations, and general citizenry. MH/MR 
board members are also encouraged to participate 
in the evaluation, but no team members may be 
paid employees of the programs evaluated. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The county mental health and mental retar- 
dation regulations specify that at least three 
services must be evaluated each year. The law 
does not specify what type of an evaluation should 
be conducted. The board felK that citizen 
evaluation teams Vv^ould be the most objective way 
of conducting evaluations on programs throughout 
the county. The evaluation also provides a means 
of citizen identification with the county programs. 

LEVEL OF PARTICIPATION 

Each fiscal year the program evaluation 
committee selects at least three types of services 
provided within Lancaster County to be evalu- 
ated. Immediate notification is sent to all 
facilities selected by the citizen committee. 
These providers also receive an explanation of the 
PASS evaluation tool and a detailed document 
explaining the evaluation process. The planner- 
evaluator, who is a staff member of the division 
of operations research, coordinates the evaluation 
process but does not take part in the evaluation 
itself. The citizen team is involved in the 
evaluation from the training workshop to the 
submission of the report to the MH/MR board. 
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TARGET OF EVALUATION 
OR MONITORING 

In 1982, nine programs were evaluated They 
included: one private outpatient facility; one 
mental health community rehabilitation program; 
a det vification unit; the administrative office of 
MH/MK, day treatment (MH) program; one drug 
and alcohol residential treatment facility; the 
drug and alcohol administration; an early 
intervintion program for mentally retarded 
children; and the consultation and education 
division of MH/MR, 

Lancaster County includes two catchment are^ 
with a combined population of 362,000 people. It 
is known for its wealth of human service 
providers. All programs that were evaluated 
rec<?ive some funding from MH/MR, although 
funding can range from several hundred to 
hundreds of thousands of dollars. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The PASS evaluation tool examines program 
administration, fiscal matters, programmatic 
issues, and the physical and social environments. 
This includes the quality, accessibility, and 
availability of the service. There have been 
occasions v^en the evaluation team used the 
FUNDET evaluation tool. FUNDET looks pri- 
marily at fiscal concerns in rsiationship to the 
needs of special groups such as blacks, children 
and youth, low-income people, Hispanics, and 
geographically isolated persons. 

TECHNIQUES USED 

After the PEC selects the agency and the 
agency is notified, the team meets to review the 
evaluation tool and also to review factual 
information about the site that is provided by the 
agency as part of a 20-page questionnaire. 
Following the information review, the team 
makes a site visit. The site visit team spends a 
day and a half observing the program and 
interviewing staff, clients, administration, 
parents, board members, and support personnel. 
The PASS evaluation is condensed into a booklet 
of worksheets 'chat the team uses throughout the 
site visit and during the reconciliation meetings. 
After the site violt, the team meets for two or 
three meetings to reconcile each of the ratings in 
PASS. Observations are made, and commendations 
and recommendations are then developed and 
placed in a report. 

FINDINGS OF EVALUATION 
OR MONITORING 

In 1982, all programs were found to be pro- 
viding adequate service, and in many cases, the 
service was found to be exceptional Site visit 
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teams felt that improvement could be made in 
areas such as case records, beautification of the 
environment, additional space and facilities, and 
changes in fiscal reporting. 

RECOMMENDATIONS 

The team addresses each finding with either a 
recommendation or a commendation. Issues a'-e 
highlighted by the team when additional concern 
should be focused uoon particular recommen- 
dations. Recommendations are directed at the 
program itself, the director, the MH/MR system, 
and also the MH/MR board. All commendations 
and recommendations are sent to the agency 
board, the director and staff, the PEC, and 
ultimately, the MH/MR board. The evaluation 
team and the agency staff meet to clarify the 
report and discuss the findings. 

STEPS TO ENSURE IMPLEMENTATION 

After the team finalizes the report, it is sent to 
the program evaluation committee and a meeting 
is held between the PEC and the agency to clarify 
the report. The PEC then develops a short report 
to the bOr.rd that is submitted with the team's 
evaluation. These are reviewed by the MH/MR 
board. The agency once again is invited to an 
MH/MR board meeting, at which time the report 
is reviewed anri recommendations made. The PEC 
report of the evaluation, with a written response 
from the agency, is then published in a public 
document and sent to the State offices of mental 
health and mental retardation, public libraries, 
the county commissioners, local college libraries, 
advocate groups such as the Lancaster Associa- 
tion of Retarded Citizens (LARC), the Mental 
Health Association, agencies throughout the 
county, and citizens who participated in the 
evaluation. The agency also must provide a 
written plan of action within 90 days of the 
MH/MR board's review of the evaluation. This 
action plan must state how the agency is to 
implement each of the recommendations. Not ali 
recommendations have to be carried out, but an 
explanation of why they are not carried out must 
be included in the report. Tne program specialists 
for mental health and mental retardation then 
monitor the implementation of these recom- 
mendations throughout the year. Agencies may be 
placed on probation or contracts terminated if the 
evaluation team and the PEC believe that the 
program does not measure up to its contract. 

EXTENT OF IMPLEMENTATION 

Most agencies that are evaluated do implement 
the recommendations. However, those recom- 
mendations requiring large amounts of money or 
that are viewed as inappropriate may not be 
implemented. Reasons for noncompliance must be 
presented in the action plan submitted to the 
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board. For example, some agencies were re- 
quested to look into the possibility of moving 
their facilities to areas that were more accessible 
to the client population. Since this recommen- 
dation would be extremely costly, the programs 
decided not to move but will take the recom- 
mendation into consideration if and when funds 
are available. Recommendations that have been 
implemented in many of the programs include 
changes in accounting procedures, beautification 
of the property, programmatic changes such as 
using generic resources throughout the com- 
munity, and many other changes that do not 
require large sums of money. 

SPECIAL BARRIERS OR SUPPORTS 

Although it is cost effective to use volunteers 
and information from the community is very 
helpful, there have been problems with volunteers 
dropping out of the evaluation. This may leave the 
team with an insufficient number of evaluators. 
The time that each evaluator must contribute to 
the process includes 1% days during the week to 
visit the site and approximately 20-30 additional 
hours to prepare the reconciliation and final 
report. Another problem is that some evaluators 
find it difficult to limit themselves to the 
evaluation tool and the team may find itself 
bogged down in extraneous topics. Using one 
standardized evaluation tool for all programs or 
agencies is sometimes a problem. The evaluation 
tool is not appropriate for all programs, although 
we find it does relate in many ways to most 
programs being evaluated. When a rating does not 
apply to a particular prog* am, the team has the 
option of stating this in the report and skipping 
that particular rating. 

Support for this evaluation process includes 
suggestions from the community for programs for 
which the community ultimately pays. The 
general public is also educated throughout the 



evaluation procbjs. This produces citizen 
advocates who are useful in other areas of the 
program and who would support the program 
financially and politically. The evaluation process 
has proved effective over the last 10 years. The 
MH/MR board has become more familiar with this 
rather large and complex system after they have 
reviewed the evaluations, and the county has 
gained a pool of dedicated and trained volunteers. 

RESOURCES AND COSTS 

Cost studies are currently being compiled, so no 
bottom line figure can be placed on the evaluation 
process at this time. The evaluations require a 
part-time secretary and a part-time evaluation 
coordinator who brings together the entire 
evaluation process. Duplication of the reports and 
support materials is another cost factor that must 
be considered. Using citizen volunteers cuts down 
on the cost of the evaluation, but there are 
overhead costs, such as food, gas, reproduction of 
materials, cost of a recognitic^ night for the 
citizens who volunteered their time. There is also 
the time and money spent by the evaluated 
agency to produce the documentation needed and 
the time spent during the interview/site visits. 
Nevert*ieless, when compared to the purchase of 
p.ofessional evaluators' time, citizen evaluation 
is cost effective. 

ADDITIONAL COMMENTS 

The PASS evaluations have been implemented 
for the past 10 years in Lancaster County and the 
office of MH/MR plans to continue using the 
instrument. The short form of PASS is currently 
being used. This reduces time and money spent on 
each evaluation. Other evaluation mechanisms are 
also being planned and implemented within 
Lancaster County in addition to the PASS citizen 
evaluations. 



50. An Agency Board Moves Toward Client Outcome Evaluation 

Vera Mellen 

The Social Center, Fairfax, Virginia 



SUMMARY 

Several years ago, the research and evaluation 
commit-tee of the board of directors of The Social 
^enter made an organized effort to obtain the 
• ta necessary to provide a basis for evaluating 
th^ center's effectiveness by tracking client 
outcomes. The initial data collection was 
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accomplished in 6 months and led to important 
changes in the recordkeeping of the agency. The 
larger goal of assessing client outcomes proved an 
ambitious aim and tnok much longer to implement. 



TYPE OF ORGANIZATION 

The Social Center is operated by a board of 
directors consisting of 20-25 citizens from the 
community. The board employs an executive 
director and a staff of 30 who serve 400 clients 
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per year. Board members are elected by the board 
upon presentation by the nominating committee. 
Every effort is made to make membership as 
representative of the community as possible. 

EVALUATORS OR MONITORS 

The research and evaluation committee of The 
Social Center's board of directors consists of 
individual board members who are lay people, 
although many have professional backgrounds in 
psychology, social work, or related disciplines. 
The membership on this committee has varied 
considerably over the several years during which 
this and related projects have operated. 



REASONS FOR THE EVALUATION 
OR MONITORING 

The lack of staff time to collect, organize, 
interpret, and utilize data was the primary 
stimulus for the initiation of the evaluation 
efforts on the part of the board. 

LEVEL OF PARTICIPATION 



The referral study was strictly an internal 
project. The evaluation committee reviewed 
referral data routinely collected by the center. 
This data had never been systematically analyzed 
by center staff, so it was up to committee 
members to determine what pieces of information 
were important for an evaluation and to design an 
analysis plan. 

TARGET OF EVALUATION 
OR MONITORING 

The Social Center is a private, nonprofit 
corporation which provides rehabilitation for 
adults with chronically severe emotional dis- 
orders. It has two locations in the Virginia suburbs 
of Washington, D.C., although at the time of this 
study the agency had three locations. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

The evaluation committee determined that 
foUowup studies of client outcome could not take 
place until more fundamental data were col- 
lected. For instance, which agencies in the 
community were making referrals? Were they 
appropriate referrals? It was also important to 
find out who was attending the center (i.e., what 
was the distribution of age. sex, race, etc.). 
Information regarding such factors as diagnosis 
and prognosis was also necessary. A long list of 
data obtainable from the agency referral form 
was collected and evaluated. The long-range goal 
was to track client outcome and agency 
effectiveness. 
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TECHNIQUES USED 

Agency referral forms for a specific fiscal year 
period were collected. All forms were examined 
by evaluation committee members and data from 
these forms were organized and a report was 
prepared. An important step was taken at this 
time. The agency's intake form was expanded to 
include a permission slip from every incoming 
client which would allow him/her to be inter- 
viewed by a member of the evaluation committee 
after he/she left the center program. This allowed 
the committee, over a period of years, to 
assemble a large body of data from which to 
organize other studies. The committee members 
were trained by staff to ensure protection of the 
confidentiality of the clients. 

FINDINGS OF EVALUATION OR MONITORING 

The referral study provided data on: numbers of 
clients, sex, employment status, diagnosis, 
appropriateness of referral, and percentage of 
referrals in which actual service was initiated. In 
addition, data among center locations were 
compared and contrasted. This information was 
critical in order for the committee to understand 
who was attending the center. Additionally, it 
provided the beginning of a database which was 
added to yearly. This result was probably the 
single most significant outcome of th" study. 

RECOMMENDATIONS 

Since this effort was not a real evaluation study 
but a collection of data, the results did not 
change the program per se. It did, however, add 
much to the understanding of staff regarding the 
flow of clients through the center. This in turn 
improved the basis upon which budget requests 
were made, etc. It also allowed for helpful 
comparisons among center locations, which gave 
rise to examinations of particularly successful 
programs in one location and attempts to 
replicate them throughout the social center sites. 
In this sense, the study provided an interesting 
evaluation tool. No actual recommendations were 
made except to redesign the intake form to allow 
followup studies of clients. 

STEPS TO ENSURE IMPLEMENTATION 

The above recommendation was implemented 
by having a staff committee redo intake forms. 

EXTENT OF IMPLEMENTATION 

The data accumulated as a result of the above 
recommendations was later compiled into a 
computerized data study under the direction of 
the research and evaluation committee of the 
board. Computer time was donated by a local 
university and the board of the social center paid 
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for the time of a programmer to code and 
program. As a result, the database was very mucn 
improved and expanded and now the center's data 
can be compared to national figures on the 
chronically mentally ill population. 

SPECIAL BARRIERS OR SUPPORTS 

The original goal proved very amL:Mous for a 
volunteer board. It has taken years to accomplish 
what originally was expected to take one or two. 
The lack of staff time also was a problem. Had 
more staff time been avaUable, the study would 
no doubt have taken less time. The commitment 
of individuals who chaired the research and 
evaluation committee seemed to make a signi- 



ficant difference regarding how much was 
actually accomplished over a given period. 

RESOURCES AND COSTS 

Accurate records of individual time given were 
not kept. The referral study itself was accom- 
plished in approximately 6 months. 

ADDITIONAL COMMENTS 

Though the study took much longer than was 
originally expected, the center expects to have an 
excellent foundation for program design and for 
the outcome studies that were the original goal of 
the committee. 



51. Citizens Use the Freedom of Information Act 

Richard M. Hessler 

University of Miscouri 
Michael J. Waiters 

Sister of Charity Hospitals, CincirmatU Ohio 



S UMMARY 

This project began as an attempt to evaluate 
followup services for adult patients discharged 
into the Mid-Missouri mental health center 
(MMMHC) catchment area. The study was 
conducted by the Mid-Missouri Citizen Evaluation 
Team (MCET). During the early stages of the 
study, the MCET was blocked by its efforts to 
obtain: (1) copies or the Mid-Missouri mental 
health center survey forms, used by the biometry 
division of the National Institute of Mental Health 
(NIMH); and (2) copies of Region VII Mental 
Health Administration's site evaluation reports on 
the mental health center. Both documents were 
considered crucial for a comprehensive evaluation 
of outreach services. This case describes and 
analyzes the process by which the MCET net the 
resistance of the Region VII office from the 
beginning of the conflict through the achievement 
of a formal amendment to the Public Information 
Regulation {Federal Register, Subpart F, Section 
5.72(d). August 17. 1973). It also describes the 
evaluation of outreach services. 

TYPE OF ORGANIZATION 

MCET members were recruited from a group of 
citizens voluntarily inv ilved in mental health 
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services in the Mid-Missouri region. MCET was 
not officially a part of the county association for 
mental health nor was it part of the citizen 
advisory board of the Mid-Missouri mental health 
center, although MCET members had a working 
relationship with these mental health agencies. 
Funding for MCET activities came from a small 
NIMH grant. The purpose of the grant was to 
assess the effectiveness of citizen evaluation. 
Team members received a $10 stipend and travel 
reimbursement for each meeting attended. The 
remaining money was available for research costs. 
Approximately 30 regular meetings were held 
with many special sessions over a 12-month 
period. A final report was written and published 
by the team memuers. 

EVALUATORS OR MONITORS 

Seven persons made up the team. Their ages 
ranged from the early twenties to late sixties. Of 
the four men and three women, five were college 
graduates and four had obtained masters degrees. 
Three members were fully employed outside the 
home, two were homemakers, one member was 
retired, and one member, employed inter- 
mittently, left the team because of a recurring 
illness. Richard Hessler and Michael Walters 
served tne dual roles of consultants and facili- 
tators to the team plus evaluation researchers 
whose goal it was to describe the effectiveness of 
citizen-controlled evaluation research. As 
researchers, Hessler and Walters recorded all 
meetings, catalogued extensive field notes, 
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memos, letters, and published two paper, in 
professional journals. 

REASONS FOR THE EVALUATION 
OR MONITORING 

The MCET members decided at their first 
meeting to explore followup services as a likely 
focus for a research problem. In order to deter- 
mine if this was an issue worth pursuing, the 
members opted for a site visit to the MMMHC. 
Existing evaluation data were sought as back- 
ground reading in preparing for the site visit. At 
the time this decision was made, only one MCET 
member knew of the existence of site evaluation 
reports but that knowledge was all it took to 
focus the data search on those reports. A quick 
check uncovered the facts that (1) a recent site 
evaluation report was done on MMMHC; (2) the 
NIMH Federal office had copies of the reports but 
preferred that MCET go to the regional office for 
copies that it had on file; and (3) the Missouri 
State division of mental health also had copies. 

The MCET members viewed the site evaluation 
reports as crucial for defining the research 
problem and they perceived their request as a test 
of citizens* access to data. The access issue was 
tied to the dual goals of MCET's substantive 
research and of testing tho citizen's right to use 
the evaluative data. The MCET request for copies 
of the report was denied by the regional office 
and a struggle ensued. 

L EVEL OF PARTICIPATION 

The MCET set as its research goal the evalua- 
tion of follo'vup services for adult mental patients 
discharged into the Mid-Missouri mental health 
center catchment area. As such, the MCET 
members had full control over all research design 
decisions, including the very definition of the 
research problem and the writing of the final 
report. It was the team's decision alone to fight 
for access to data defined by team members to be 
essential. 

TARGET OF EVALUATION 
OR MONITORII J G 

The Mid-Missouri mental health center is 
located in a rural area, near Tipton, Missouri. This 
study focused on outreach services to clients 
residing in MMMHC's catchment area. 

PROBLEMS OR ISSUES EVALUATED 
OR MONITORED 

Originally, the target evaluation was 
followup services provic' to adult patients 
discharged to the catchment area served by the 
Mid-Missouri n>ental health center. When team 
members were unable to obtain copies of NIMH 
site visit reports, the focus quickly changed to an 
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attempt by private citizens to gain access to 
evaluation information compiled on a publicly 
funded agency. 

The MCET and NIMH represent two insti- 
tutional viewpoints in conflict over interpre- 
tations of their respective mandates or claims. 
The NIMH regional office defined access to 
biometry data as limited only to ••professionals'* 
whereas the MCET members thought that this was 
elitist. Neither side could see any merit in the 
other's claim and an impasse was reached very 
quickly. 

The beginning of the conflict had distinct and 
very different features than its later phases. At 
first, the MCET members were unsure of who was 
accountable for the decision to bar access to the 
site visit documents. They had a limited under- 
standing of the substantive worth of the forbidden 
data, and they were reluctant to firmly insist that 
the data be made available for fear of jeopar- 
dizing the research itself. Similarly, the other 
side was uncertain about the competence and 
tenacity of the MCET members, the legitimacy of 
the proposed research, and their own legal 
position. MCET was committed to defining the 
research problem and the site evaluation data 
were viewed as an indispensable aid. The NIMH 
regional office was equally committed to keeping 
the site evaluation data out of the MCET 
members' hands. 

TECHNIQUES USED 

MCET's strategy was based on a desire to avoid 
contaminating the research on followup services. 
The team members felt that pressure placed on 
the center's administration would have negative 
consequences in terms of gaining cooperation 
from the center staff to do interviews concerning 
followup services. From the MCET perspective, it 
was more desirable to confront the State division 
of mental health and the regional office of NIMH 
than it would have been to fight the battle closer 
to home. 

The NIMH regional office staff also wanted to 
keep things under control, and attempted to 
constrain MCET members in their efforts to 
obtain the necessary documents. The strategy 
involved, (1) informing MCET that the State 
division of mental health (grantee for MMMHC) 
was the appropriate source for procuring the site 
evaluation report; and (2) claiming that they had 
been advised by the NIMH Federal office not to 
release site evaluaticn reports because they were 
exempt from disclosure under section 5.75 and 
example 4 of appendix A of tf.s Public Infor- 
mation Act. Thus, the NIMH regional office staff 
sought to redirect MCET's challenge by claiming 
powerlessness to act and by passing the buck to 
another institution. The outcome was that neither 
the Federal nor the regional office produced the 
evaluation report. 

The MCET compiled and sent a letter to the 
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state mental health office. Once again, MCET's 
legitimacy was questioned as part of the reason 
for rejecting the request. The MCET members 
decided to meet with the State director and to 
proclaim their legitimate role a'? researchers. 
Simultaneously, MCET members icquested a 
meeting with regional office administrators for 
the same reason. 

A phone call to the regional office to set up the 
proposed meeting led to the following: (1) regional 
office administrator: did not want to meet with 
MCET members; (2) a regional office lawyer 
added a legal dimension to the refusal based on 
section 5.75 and example 4 of appendix A, Public 
Information Act; and (3) regional office adminis- 
trators reiterated that it was "the responsibility 
of the Federal office of NIMH to supply you with 
tne needed data." The reasoning here was that 
since NIMH funded MCET, NIMH should provide 
the needed data. Two days later, another phone 
call, this time from the administrator at the 
NIMH regional office, added the following 
boundaries to the refusal: (1) Citizen evaluators 
did not have a legitimate role to play in 
(a) evaluating a center's compliance with the 
terms of a contract or (b) in developing policy 
through research; (2) Citizens should assist the 
regional office in developing services at 
centers— period; (3) MCET should try to negotiate 
an "exception to the law" by contacting the chief 
of the legislative services branch, NIMH, and 
HEW Assistant Secretary of Public Affairs. The 
site evaluation report had become an end unto 
itself, a symbol of power and control for both 
sides, the perceived importance of which far 
exceeded its potential substantive utility. 

The MCET members wrote a letter to the chief 
of the NIMH legislative services branch describing 
the status of the refusal. Simultaneously, the 
MCET project officer at NIMH was phoned and he 
informed the MCET that the regional office 
administrator had asked him to intercede. The 
project officer told MCET that he refused. 
Furthermore, he offered to get an endorsement 
for MCET from the section on citizen partici- 
pation of the American Psychological Association. 

The pathway to a legal resolution was now in 
place. The MCET chairman wrote a letter to the 
assistant chief of the legislative services branch, 
NIMH, laying out all that had transpired and asked 
for a formal, precedent-setting amendment to the 
Public Information Act that would enable citizen 
groups to gain access to site evaluations of 
mental health centers. The assistant chief wrote 
back to inform the MCET members that he ho J 
directed the HEW Freedom of Information officer 
to rule on MCET's case. 

Slightly less than 1 month after the MCET 
chairman wrote asking for an amendment to the 
Public Information Act, the assistant chief of the 
legislative services branch responded to the 
MCET chairman informing him that the regional 
office must release the reports lo the MCET. 
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More importantly, his ruling reflected a new 
amendment to the Public Information Act. This 
new amendment guaranteed MCET, as well as 
other citizen groups, access to these and similar 
reports. 

Once the MCET members obtained the reports 
needed, they moved on to the evaluation of 
outreach services using the NIMH reports for 
background. The evaluation of outreach services 
'•ncluded interviews with the mental health center 
director, staff of the outreach service, staff in 
the NIMH regional office, the Missouri director of 
mental health, and staff from other social service 
agencies who might have contact with clients in 
the community. 

FINDINGS OF EVALUATION 
OR MONITORING 

Once the data were reviewed, the MCET 
members concluded that outreach services were 
fragmented and that many clients were "falling 
between the cracks." FoUowup services were 
frequently deficient and many clients were 
floundering in the community. 

RECOMMENDATIONS 

The MCET recommended that foUowup services 
needed to be improved and that a system for 
keeping track of clients in the community should 
be implemented. 

STEPS TO ENSURE IMPLEMENTATION 

The MCET members presented their findings to 
the center and used the report to gain community 
support for the development of a halfway house 
for women who were being dischaged from the 
hospital. 

EXTENT OF IMPLEMENTATION 

With the exception of the halfway house, the 
MCET's recommendations were not implemented 
because funding was not available. 

SPECIAL BARRIERS OR SUPPORTS 

The major barrier in conducting this sti;ciy was 
the resistance that MCET members encountered 
in their attempts to gain access to reports. The 
major support was the perseverance of the team 
members. 

RESOURCES AND COSTS 

Ti.is project was funded as part of a larger 
NIMH team. The team's budget was approxi- 
mately $4,600 and covered the cost of transpor- 
tation, printing, postage, etc. Team members 
were also paid a small honcr<arium for attending 
meetings. 
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ADDITIONAL COMMENTS 

The MCET victory was the outcome of a 
protracted struggle between two institutions 
vying for legitimate control over evaluation data. 
The NIMH regional office sought to protect its 
claim of professional competence by defining the 
citizen evaluators as Imposters, as perpetrators of 
the erroneous Idea that individuals other than 
mental nealth professionals can manage the 
analysis of complex evaluation data, including the 
protection of respondents' anonymity and other 
commitments made by the professionals. The 



MCET members used the site evaluation data as a 
symbol of their legitimate role as researchers. By 
achieving possession of the data, the claim to 
professional competence was established along 
with the legal mandate that institutionalized the 
legit'^nacy of the citizen evaluation research role. 

The MCET case heralded a genuine shift in 
power, not some ritualized demonstration project. 
The nrecedent for future citizen evaluative 
efforts was set, the knowledge of "how to do It** 
was documented, and the mental health discipline 
awaits other efforts of citizens to improve 
services through research. 
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How WOl the Results Be Used? 



Doug Brown and his evaluation committee reviewed available county 
and State data and conducted interviews with administrators from the de- 
partment of mental health and mental retardation, officials from River- 
view State Hospital, and providers from the community service system. 
After analyzing and synthesizing the information, the committee compiled 
a draft report. The draft was submitted to the mental health association, 
the county and State mental health departments, Riverview administrators, 
and the community providers who had been interviewed. The comments 
provided by all parties were reviewed and, where appropriate, changes 
were made to the final version of the report. 

Two major findings emerged from the stud". One was that there ap- 
peared to be a history of poor communication b. :v/c6n the county depart- 
ment of mental health and Riverview State Hospital. The committee mem- 
bers feared that this poor communication could jeopardize the orderly 
movement of clients from the institution cO the community. Secondly, the 
commit' ee found that the community was ill prepared for any large de- 
institutionalization. They found inadequacies in community services for the 
elderly, in housing, and in aftercare programs. 

Based on these findings the committee made two recommendations. 
First, they suggested that ore person at the county department of mental 
health smd one person at Ri view should be appointed as liaison persons 
to oversee the process of closing the Riverview wing and to ensure ade- 
quate communication. Secondly, the committee recommended that clients 
from the unit should be discloarged in stages over a 2-year period so that 
the community service sjrstem would have time to gear up to provide an 
adequate level of service. 

The committee felt strongly about the importance of these recom- 
mendations. They therefore spent considerable time on their strategy to 

w&wvk* \^ v»»<Av v»»\^ A v/v/vr»»iA»Av^A»vM4'Vi.v»i*> w%/U>lv* K/K^ UiiJ^lCi iiCiiCCU. lllCil 1U1>L SLCSp 

was to debrief the State and county officials of the departments of mental 
health. They communicated their findings, voiced their concerns, and at- 
tempted to get a commitment from their departments. Secondly, they 
contacted their local State representative and communicated the findings 
of the study to her. She arranged for Doug Brown, as the committee 
chairman, to make a presentation before the legislative btadget committee. 
Finally, the committee wrote a press release, and the study's findings and 
recommendations were disseminated through the local media. 



This chapter describes some of the techniques 
that citizen evaluators cam use to ensure the 
visibility of evaluation and monitoring results and 
the ultimate implementation of recommendations. 
The failure to give adequate consideration to a 
strategy for disseminating study results and for 
maximizing the chances that recommendations 
will be implemented is a serious shortcoming of 
many evaluation studies conducted by professional 
as well as citizen evaluators. In conducting a: - 
sessments of services, evaluators are often 
strongly tempted to focus too much eneri^y O) ^'le 
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process of the study itself. No matter how well 
the evaluation process is carried ou* . it is unlikely 
that the study will have its intended effect with- 
out a proportional escpenditure of effort on 
dissemination. 

Decisions about poss'Mt implementation 
strategies are constrained to some extent by the 
group'?- position or its relationship to the mental 
health system. In some caues, groups have formal 
authority and can demand that their recom- 
mendations be implemented. Other groups have to 
be more creative to make their points. In this 
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chapter, we 'mmmarize some of the strategies 
that gr ups might use by referring back to cases 
discussed in the preceding chapters. 

Agency Debriefing 

Three cases illustrate -the use of agency de- 
briefings. Agency debriefing means simply that 
the evaluators or monitors communicate their 
findings and recommendations directly to the 
agency or agencies evaluated or monitored. This 
is nearly always good practice. In the first place, 
agency staff frequently are ir the best position to 
implement study recommendations. Secondly, it is 
bad politics to pass study results on to county or 
State officials without notifying the agencies and 
giving them the opportunity to respond. 

One example of this approach is described in 
case #45 by Aldene Hart and Barbara Goza (Cit- 
izens* Advisory Council Uses Case Simulation 
Technique to Evaluation Crisis Services). In this 
case, a citizens* advisory council evaluated the 
quality and accessibility of a CMHC's crisis 
services. After completing the study, citizens 
presented the study findings to the CMHC staff 
and commended them for the high-quality service 
they were providing. Giving agencies this type of 
positive feedback is a useful way of enhancing the 
relationship between the citizen group and local 
service providers. 

Another example of agency debriefings is de- 
scribed in case #49 by Rick Kastner and Marilyn 
Lee Olds (Citizen Evaluation Using Program An- 
alysis of Service systems). In this case, the re- 
searchers met *^th staff from the various agen- 
cies evaluated to review the study findings. After 
the agency had the opportunity to respond, the 
report was forwarded to the county mental health 
and mental retardation board. Agency staff were 

report was discussed. 

A third example, case #20, (Council for Com- 
munity Services Agency Evaluation) by Beverly 
Kries et al., describes how evaluators can involve 
agency staff throughout the entire evaluation 
process. The task force conducting the evaluation 
provided feedback and technical assistance as 
issues emerged during the evaluation process. As 
a result, the authors report, many of the study's 
reccnrmiendations had already been implemented 
by the time the evaluation was completed. 

Inclusion in an Annual Plan 

Frequently, boards have the authority to in- 
corporate their recommendations into the annual 
plan of an agency, a county, or a State. This is a 
useful implementation strategy since it estab- 
lishes a benchmark against which the next year's 
performance can be measured. Several of the 
previous cases exemplify this type of implemen- 
tation strategy. 

The authors of two cases describe agency 



boards that have used this strategy. Roger Strauss 
(case #7, Mental Health Center Board Increases 
Involvement in Evaluation of Center Needs) de- 
scribes a njeds assessment conducted by a center 
board in Vermont, and Donald Eib (case #11, 
Citizens as Subjects of Evaluation) describes a 
project involving the de^'^^lopment of a model for 
long-range planning. In both cases, the recom- 
mendations resulting from the studies were in- 
corporated into the agency's annual plan. 

At the county level, Jean Fiore (case #16, 
C"* nty Board Assists in Assessing Local Services) 
describes a county board's evaluation of emer- 
gency services. In order to ensure that their re- 
commendations were addressed, the board's study 
committee disseminated its final report to the 
full county advisory board, the chairperson of the 
county department of human services, and the 
State regional office. They also included their 
reconunendations in the county plan. 

Dennis Geersten (case # 1 , State Council 
Assesses Utah's Mental Health Needs) describes 
this same process at the same level. In this case, 
Utah's State planning and advisory coracil con- 
ducted a needs assessment of the problems of the 
State's special populations and their service 
needs. The board's recommendations were in- 
corporated into the State's annual plan. 

Securing Political Support 

Another strategy that may be useful on oc~ 
casion is enlisting the support of State legislators. 
Jan Holcomb et al., in case #30, describe how a 
mental health association cultivated relationships 
with legislators and their staff. This is a useful 
technique that brings necessary information to 
the groups that control the allocation of re- 
sources. This case report describes how asso- 
ciation members assisted State legislators in 
developing an agenda for public hearings on ap- 
propriations for mental disabilities, offered tes- 
trniony to legislative committees, and worked 
vdth their association lobbyist to introduce leg- 
islation recommending changes in the State hos- 
pital system. 

Constituency Building/Information Sharing 

Constituency building is another strategy that 
can be used to disseminate study results and gain 
support for recommendations. Frequently, gaining 
the support of other civic or community groups 
can increase the credibility of one's arguments. 
This method is illustrated in case #32, described 
by Frances P. Meehan (Citizens Evaluate Alio- 
cation of County Mei*cal Health Resources). In 
this case, a citizen review of county data re- 
vealed that county mental health funds appeared 
to be distributed unevenly across different regions 
of the county. Members of the mental health 
services liaison committee presented their find- 
ings to a wide variety of civic groups in order to 
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gain popular support for their case. The groups 
included the mental health advisory board, the 
American Association of University Women, the 
League of Women Voters, the regional planning 
coiancD of the United Way, and a variety of 
service clubs. In the end, the liaison committee 
was able to enlist the support of 12 community 
leaders who joined the committee in a joint pre- 
sentation to the area's county supervisor. 

Funding Sanctions 

The next stra' ;gy to be discussed is the use of 
funcUng sanctions. Clearly, the threat of with- 
holding funds cannot be used by groups whose 
power is limited to moral suasion or the like. 
However, for those groupj that have some amoxint 
of authority over the funding process, this is a 
poweaul tool indeed. In case *3, Je'^sica Wolf 
describes an evaluation of an agency halfway 
house and supervised apsirtment program. Con- 
necticut law requires that agencies be evaluated 
by regional mental health boards. These boards 
have the authority to either approve or disapprove 
continued funding of mental health programs. In 
this case, the initial recommendation to the re- 
gional board was to terminate funding for the 
apartment program because of low occupancy 
rates. After reviewing additional data suggesting 
an improved occupancy rate, the board recom- 
mended conditional funding. 

Diane Rich (case #15, Citizen Participation in 
Federal-State Site Visits) describes how a mental 
health association's recommendations were in- 
cluded in Federal reports that ultimately affected 
whether agencies received continued fmding. 
Similar sanctions are described in case #44 by 
Frances P. Meehan and John J. McDonough (A 
Mental Health Advisory Board's Use of Secondary 

Rick Kastner and Marilyn Lee Olds (Citizen 
Evaluation Using Program Analysis of Services 
Systems). 

Use of the Media 

For those groups that do not have authority 
over funding, an alternative strategy might be to 
bring the czise to the media. This strategy v/as 
utilized quite effectively by members of the 
Mental Health Association of Greater Chicago 
(Jan Holcomb et al., case #30). In this instance, 
members of a site- visit team suggested to re- 
porters from the Chicago Sun Times that they 
expose some of the systemwide problems in health 
services. This led to a six- part newspaper series 
entitled "Breakdown: Mental Health in Crisis." As 
a result of this publicity, the Governor became 



involved in the issue, appointed a task force, and 
invited the association's participation in the men- 
tal health planning process. Barbara Geddie (case 
#43, Emergency Services in Hi^ Point) also de- 
scribes the use of ra<tio, television, and news- 
papers in a local mental health association's 
attempts to publicize the need for afterhours 
mental health emergency services. 

Litigation or Otner Advocacy 

A very powerful implementation strategy that 
probably should be used only as a last resort is the 
use of the courts. Darlene Humphrey (case #5, 
External Group Advocates for Better Care) de- 
scribes the efforts of Consumer Advocates for 
Better Care (CABC) to improve conditions in 
nursing homes, CABC monitors nursing homes 
through site visits ^nd informal interviev/s wth 
residents. The case described by Ms. Humphrey 
involved a nursing home that was cited for a 
number of serious abuses. CABC turned its find- 
ings over to a local legal-services agency which in 
turn passed the information to the State attorney 
general's office. As a result of these actions, thi^ 
particular rest home has been closed and its li- 
cense suspended. 

As this case illustrates, litigation is an ex> 
tremely powerful strategy. However, it is time- 
consuming, costly, and tends to have the inevi- 
table byproduct of polarizing both sides. If a 
citizen group must rv y on the target of evalua- 
tion for cooperation over the long term, then 
some other means of dispute resolution should be 
attempted first. 

Stacking the Board 

Finally, under the heading of "If you can't lick 
liiciii , . ." 16 liic Iiii^iciiic'iUctuuii 5I1 dtc^ uf 
placing citizen evaluators on agency boards or 
higher level boards to ensure a continued pres- 
ence. Rita Parle (case #18) describes a com- 
munity group's evaluation of a mental health 
center that had been the object of controversy 
wthin the community. After completing an 
evaluation that led to numerous recommen- 
dations, the external committee attended a num- 
ber of board meetings to ensure that the concerns 
raised in its report were addressed. In addition, 
one member wais appointed to the center's board 
of trustees and several other members joined the 
center's advisory board. 

A variation of this is for the board to hire or 
recommend that the agency hire one or more in- 
dividuals for the purpose of implementing the 
study's recommendations. This approach is il- 
lustrated in case #35 ty Joyce G. Smith. 
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Summary 



The cases presented in this Casebook demon- 
strate the inventiveness with which citizens have 
approached evaluation and monitoring tasks and 
the quality of the work that has resulted. The 
contributions of citizens involved in evaluation 
and monitoring have been significant; however, 
there are certain concerns and issues that should 
be highlighted if such involvemert is to continue 
as a strategy for improved mental health and 
other human services. 

On the positive side, citizen involvement in 
evaluation has deiinite advantages for the citizen 
group or organization, for the individual citizen, 
and for the agency or agencies being examined. 
Citizen involvement in evaluation frequently re- 
sults in increased visibility and credibility for the 
citizen group or association. Over time, it also 
generates for the association a pool of individuals 
with the research skills necessary to conduct new 
studies or to review critically the reports gener- 
ated by professionals within the system. 

For the individual citizen, the experience also 
appears to have positive results. Citizens report a 
feeling of accomplishment once the projects are 
completed, and they appreciate the new skills 
acquired during the process. Some citizens have 
oven expanded their ovsn personal development by 
obtaining more education or moving into more 
visible positions in the community. The experi- 
ence also provides them with increased knowledge 
about tlie local system. 

From the agency's perspective, one over- 
whelming adv2Lntage is the economy involved In 
lising citizens as evaluators and monitors. With 
increasing budget constraints, this factor assumes 
even greater importance. The agency also bene- 
fits from the different perspective that citizens 
who are net formally part of the system bring to 
the evaluation or monitoring process. Finally, to 
the extent that involvement in evaluation or 



monitoring imparts to citizens a sense of owner- 
ship of the program or programs, the agency also 
benefits from the political support that such ad- 
vocates can muster. 

Although citizen involvement in evaluation and 
monitoring has very clear advantages, there are 
also some drawbacks. One is that the process can 
at times be inefficient due to the inexperience of 
the individuals involved. Citii^ens very often re- 
quire training in evaluation techniques, in the use 
of survey interview forms, and in the general 
conduct of program review activif'es. Secondly, 
citizens may have significant constraints on their 
time and almost always have little in the way of 
financial resources for projects. Moreover, vol- 
unteers may relocate or become employed, thias 
providing little continuity in implementing the 
evaluations. Also, because of their position within 
the system, they fre:: ^^tly have little authority 
to demand access to iiiformation and mtist rely on 
moral suasion instead. 

Finally, since evaluation and monitoring have 
traditionally been conducted by trained profes- 
sionals, agencies are frequently wary of allowing 
citizens who have little formal training to take 
over these tasks. Agencies may simply resist ai^y 
review by the community, espenally if program 
inadequacies have already been highlighted by 
others. Citizens, however, can minimize agency 
resistance by identifying key staff who are willing 
to cooperate with and support outside evaluators 
or monitors. One or iiiurp lidcuial ouvocatos will 
ease implementation of both citizen evaluation 
activities and the ensuing^ results. 

These drawbacks notwithstanding, the cases 
described in the Casebook demonstrate clearly 
that citizen groups are capable of conducting 
skillful and innovative evaluation projects, and 
that these projects can have an impact on the 
mental healtix system. 
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Giossaiy of Terms 



Balanced service system- -aTi accreditation scheme developed in the 1970s 
by the Joint Commission on the Accreditation of Hospitals to assess the 
quality of services delivered by CMHCs. The standards reflected both 
\Taditional and nontraditional service delivery approaches. 

Barrier free — describes an en\ironment such as a house or office building 
which is accessible to physically handicapped persons. For example, door- 
ways may be widened to accommodate wheelchairs, or ramps might 
replace stairways. 

Case management- -an approach to organizing service delivery in which 
one individual -a case manager— has r-^sponsibility for coordinating all of 
tho services required by an individual client. 

Catchment area- -the geographic area for which a community mental 
health center has responsibility. The population size of catchment ar^^as 
generally ranges from 75,000 to 200,000 persons. 

Cost effectiveness- -maximiziiig the achievement of program goals (e.g., 
improving client outcomes) v/hUe minimizing program costs. 

Cost e//fcfeney- -maximizing output (e-g., increasing the number of 
clients served) while minimizing the expenditure of resources. 

Diagnostic Statistical Manual III (DSM ///>— diagnostic code published by 
the American Psychiatric Association in 1981. It is a standard taxonomy 
of diagnoses and is used internationally. 

Field test--vAx^xi evaluators design a new instrument, they typically try it 
out on a few respondents in order to identify any problems with the in- 
strument itself. This is also referred to as a pretest. 

Halfway house- a residence desigiied for individuals making the transition 
from a hospital setting to community living. 

Inservice training— semmar and workshop programs that are offered by 
employers. The purpose of inservice training is to help staff improve their 
profe^ional skills and to keep staff informed of .lew developments in 
their fields. 

Key informant' knowledgeable individuals within the community who can 
provide data or interpret events that are relevant to the particular 
project. 

Needs assessment- -a study that attempts to identify the service needs of 
populations or special subgroups. A nee4s assessment may also include an 
examination of sei vices that are already in place and an identification of 
service gaps. 

Nominal Group Technique—a structured group approach to problemsolving 
in which equal participation of all group members is solicited. 
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Partial hospitalization- -programs designed for mentally ill clients who 
need more than traditional outpatient therapy but do not require inpatient 
hospitalization. Clients typically attend partial hospital programs for ha^f 
or full day sessions during which they receive a variety of mental health, 
social, and vocational services. These programs are sometimes referred to 
as day treatment programs. 

Psychosocial reJiabilitation center- ■ an agency providing traditional 
mental health services as well as a variety of social leamuig, vocational, 
and community living programs. 

Psychotropic medications' -^ixug^ that are prescribed by physicians to 
control some of the symptoms of mental illness. 

Reliability" the accuracy of a measure over repeated trials. 

Sample— a subset of cases selected for study from a larger population. A 
good sample is representative of the population from which it was drawn 
and the sample data can, therefore, be used to describe the larger popu- 
lation. Sampling is used because it is frequently tco costly to collect data 
on all relevant cases in a population. 

Secondary data analysis- analyses that rely on data that have been col- 
lected for another purpose but are available and relevant to the particular 
study. 

Transitional housing- -hovismg programs that are designed for clients mov- 
ing out of a hospital setting. These programs provide supervised living 
arrangements and typically offer a variety of other mental health and 
community support services. 

Validity- \he degree to which an indicator actually meaures what it is 
intended to measure. 
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